What happens when your walls begin to move? This paper explores the design of human-robot interaction for architectural-scale, shape-changing environments. We present findings from two studies: (1) a series of speculative design workshops (N=20) that uncovered aspirational visions for these spaces, and (2) a task-based Wizard-of-Oz elicitation study (N=12) that grounded these visions in the challenges of practical interaction. Our workshop findings reveal a complex landscape of user desires, exposing critical tensions between proactive automation and the preservation of user autonomy, and between personalization and public ownership. Our elicitation study reveals a set of core interaction challenges related to multimodal collaboration; and, most critically: suggests the need for a modality-agnostic model of evolving user intent. We conclude with a set of grounded proposals for creating robotic environments that are collaborative and trusted partners in everyday life.

It can be hard to design a physical structure entirely within the confines of a computer monitor. To better capture the interplay between real-world objects and a designer's work-in-progress, practitioners will often go through a sequence of low-fidelity prototypes (paper, clay, foam) before arriving at a form that satisfies both functional and aesthetic concerns. While necessary, this model-making process can be quite time-consuming, particularly at larger scales, and the resulting geometry can be difficult to translate into a CAD environment, where it will be further refined. This paper introduces a user-adjustable, room-scale, "shape-aware" mesh structure for low-fidelity prototyping. A user physically manipulates the mesh by lengthening and shortening the edges, altering the overall curvature and sculpting coarse forms. The edges are equipped with resistive length sensors, and transmit their configuration to a central computer. The structure can later be reproduced in software, connecting this prototyping stage to the larger computational design pipeline.

We propose augmenting initially passive structures built from simple repeated cells, with novel active units to enable dynamic, shape-changing, and robotic applications. Inspired by metamaterials that can employ mechanisms, we build a framework that allows users to configure cells of this passive structure to allow it to perform complex tasks. A key benefit is that our structures can be repeatedly (re)configured by users inserting our configuration units to turn the passive material into, e.g., locomotion robots, integrated motion platforms, or interactive interfaces, as we demonstrate in this paper.

By documenting and annotating one author's ongoing project combining 3D printing and handweaving to produce computational hybrid baskets, we contribute a framework for understanding hybrid craft. We identify three levels of material practice as observed in the basketry project—physical joinery between rigid printed-plastic parts and soft textiles, seamful multipart fabrication workflows, and aesthetics which negotiate between “basketlike” and “computational” forms—and analyze tensions and possible resolutions at each level.

The widespread adoption of 3D printers exacerbates existing environmental challenges as these machines increase energy consumption, waste output, and the use of plastics. Material choice for 3D printing is tightly connected to these challenges, and as such researchers and designers are exploring sustainable alternatives. Building on these efforts, this work explores using spent coffee grounds as a sustainable material for prototyping with 3D printing. This material, in addition to being compostable and recyclable, can be easily made and printed at home. We describe the material in detail, including the process of making it from readily available ingredients, its material characteristics and its printing parameters. We then explore how it can support sustainable prototyping practices as well as HCI applications. In reflecting on our design process, we discuss challenges and opportunities for the HCI community to support sustainable prototyping and personal fabrication. We conclude with a set of design considerations for others to weigh when exploring sustainable materials for 3D printing and prototyping.

Computational mediation can unlock access to existing creative fabrication tools. By outfitting an otherwise purely mechanical hand-operated knitting machine with lightweight sensing capabilities, we produced a system which provides immediate feedback about the state and affordances of the underlying knitting machine. We describe our technical implementation, show modular interface applications which center the particular patterning capabilities of this kind of machine knitting, and discuss user experiences with interactive hybrid computational/mechanical systems.

We propose an approach to enabling exploratory creativity in digital fabrication through the use of grain spaces. In material processes, “grain” describes underlying physical properties like the orientation of cellulose fibers in wood that, in aggregate, affect fabrication concerns (such as directional cutting) and outcomes (such as axes of strength and visual effects). Extending this into the realm of computational fabrication, grain spaces define a curated set of mid-level material properties as well as the underlying low-level fabrication processes needed to produce them. We specify a grain space for computational brioche knitting, use it to guide our production of a set of hybrid digital/physical tools to support quick and playful exploration of this space’s unique design affordances, and reflect on the role of such tools in creative practice.

For non-technical domain experts and designers it can be a substantial challenge to create designs that meet domain specific goals. This presents an opportunity to create specialized tools that produce optimized designs in the domain. However, implementing domain-specific optimization methods requires a rare combination of programming and domain expertise. Creating flexible design tools with re-configurable optimizers that can tackle a variety of problems in a domain requires even more domain and programming expertise. We present OPTIMISM, a toolkit which enables programmers and domain experts to collaboratively implement an optimization component of design tools. OPTIMISM supports the implementation of metaheuristic optimization methods by factoring them into easy to implement and reuse components: objectives that measure desirable qualities in the domain, modifiers which make useful changes to designs, design and modifier selectors which determine how the optimizer steps through the search space, and stopping criteria that determine when to return results. Implementing optimizers with OPTIMISM shifts the burden of domain expertise from programmers to domain experts.

Robotic surfaces, whose form and function are under computational control, offer exciting new possibilities for environments that can be customized to fit user-specific needs. When these surfaces can be reprogrammed, a once-static structure can be repurposed to serve multiple different roles over time. In this paper, we introduce such a system. This is an architectural-scale robotic surface, which is able to begin in a neutral state, assume a desired functional shape, and later return to its neutral (flat) position. The surface can then assume a completely different functional shape, all under program control.

Knitting machines can fabricate complex fabric structures using robust industrial fabrication machines. However, machine knitting’s full capabilities are only available through low-level programming languages that operate on individual machine operations. We present KnitScript, a domain-specific machine knitting scripting language that supports computationally driven knitting designs. KnitScript provides a comprehensive virtual model of knitting machines, giving access to machine-level capabilities as they are needed while automating a variety of tedious and error-prone details. Programmers can extend KnitScript with Python programs to create more complex programs and user interfaces. We evaluate the expressivity of KnitScript through a user study where nine machine knitters used KnitScript code to modify knitting patterns. We demonstrate the capabilities of KnitScript through three demonstrations where we create: a program for generating knitted figures of randomized trees, a parameterized hat template that can be modified with accessibility features, and a pattern for a parametric mixed-material lampshade. KnitScript advances the state of machine-knitting research by providing a platform to develop and share complex knitting algorithms, design tools, and patterns. 1

The Internet-of-Things (IoT) promises to enhance everyday objects with computing, but rarely enables directly authoring or composing that behavior. Lightweight IoT approaches attach identifiers (e.g., RFID tags) to objects to enable networked services. Typically these tags are passive, and so, depend on activity recognition and predefined context. This limits interaction to invoking predetermined behavior. Instead, this work presents The IoT Codex: a lightweight approach to customizing everyday objects with IoT by enabling interactive attachable IDs (aIDs) to compose software-supported behavior in situ. This work contributes 1) paper engineering techniques to construct aIDs that embody state, and 2) a tangible, end user programming (EUP) language for customizing IoT within symbolic and idiosyncratic contexts. Here, we provide preliminary validation of our approach with an empirically informed design space, sample applications, and a small co-design workshop. In doing so, we offer preliminary evidence for tangible, end user programming to enable meaningful control over IoT services.

The U.S. National Institute of Health (NIH) 3D Print Exchange is a public, open-source repository for 3D printable medical device designs with contributions from clinicians, expert-amateur makers, and people from industry and academia. In response to the COVID-19 pandemic, the NIH formed a collection to foster submissions of low-cost, locally manufacturable personal protective equipment (PPE) . We evaluated the 623 submissions in this collection to understand: what makers contributed, how they were made, who made them, and key characteristics of their designs. We found an immediate design convergence to manufacturing-focused remixes of a few initial designs affiliated with NIH partners and major for-profit groups. The NIH worked to review safe, effective designs but was overloaded by manufacturing-focused design adaptations. Our work contributes insights into: the outcomes of distributed, community-based medical making; the features that the community accepted as “safe” making; and how platforms can support regulated maker activities in high-risk domains.

Tactile maps can help people who are blind or have low-vision navigate and familiarize themselves with unfamiliar locations. Ideally, tactile maps can be customized to an individual’s unique needs and abilities because of their limited space for representation. We present Maptimizer, a tool that generates tactile maps based on users’ preferences and requirements. Maptimizer uses a two stage optimization process to pair representations with geographic information and tune those representations to present that information more clearly. In a small user study, Maptimizer helped participants more successfully and efficiently identify locations of interest in unknown areas. These results demonstrate the utility of optimization techniques and generative design in complex accessibility domains.

We present a framework for gesture customization requiring minimal examples from users, all without degrading the performance of existing gesture sets. To achieve this, we first deployed a large-scale study (N=500+) to collect data and train an accelerometer-gyroscope recognition model with a cross-user accuracy of 95.7% and a false-positive rate of 0.6 per hour when tested on everyday non-gesture data. Next, we design a few-shot learning framework which derives a lightweight model from our pre-trained model, enabling knowledge transfer without performance degradation. We validate our approach through a user study (N=20) examining on-device customization from 12 new gestures, resulting in an average accuracy of 55.3%, 83.1%, and 87.2% on using one, three, or five shots when adding a new gesture, while maintaining the same recognition accuracy and false-positive rate from the pre-existing gesture set. We further evaluate the usability of our real-time implementation with a user experience study (N=20). Our results highlight the effectiveness, learnability, and usability of our customization framework. Our approach paves the way for a future where users are no longer bound to pre-existing gestures, freeing them to creatively introduce new gestures tailored to their preferences and abilities.

We present an ethnographic study of a maker community that conducted safety-driven medical making to deliver over 80,000 devices for use at medical facilities in response to the COVID-19 pandemic. To achieve this, the community had to balance their clinical value of safety with the maker value of broadened participation in design and production. We analyse their struggles and achievement through the artifacts they produced and the labors of key facilitators between diverse community members. Based on this analysis we provide insights into how medical maker communities, which are necessarily risk-averse and safety-oriented, can still support makers' grassroots efforts to care for their communities. Based on these findings, we recommend that design tools enable adaptation to a wider set of domains, rather than exclusively presenting information relevant to manufacturing. Further, we call for future work on the portability of designs across different types of printers which could enable broader participation in future maker efforts at this scale.

Machine knitting is an increasingly accessible fabrication technology for producing custom soft goods. However, recent machine knitting research has focused on knit shaping, or on adapting hand-knitting patterns. We explore a capability unique to machine knitting: producing multilayer spacer fabrics. These fabrics consist of two face layers connected by a monofilament filler yarn which gives the structure stiffness and volume. We show how to vary knit patterning and yarn parameters in spacer fabrics to produce tactile materials with embedded functionality for forming soft actuated mechanisms and sensors with tunable density, stiffness, material bias, and bristle properties. These soft mechanisms can be rapidly produced on a computationally-controlled v-bed knitting machine and integrated directly into soft objects.

We present an inexpensive tabletop loom that offers fully computational patterning while maintaining the flexibility of handweaving. Our loom can be assembled for under US$200 with 3D printed parts, and it can be controlled straightforwardly over USB. Our loom is explicitly a hand loom: that is, a weaver is required to operate the weaving process and may mediate row-by-row patterning and material specifics like yarn tension. Our approach combines the flexibility of fully analog handweaving with the computational affordances of digital fabrication: it enables the incorporation of special techniques and materials, as well as allowing for the possibility of computational and creative interventions in the weaving process itself. In taking this approach, we aim to serve a range of end users including artisans and researchers, whether for skill-building, for rapid prototyping, or for creative reflection.

Knitting is a popular craft that can be used to create customized fabric objects such as household items, clothing and toys. Additionally, many knitters find knitting to be a relaxing and calming exercise. Little is known about how disabled knitters use and benefit from knitting, and what accessibility solutions and challenges they create and encounter. We conducted interviews with 16 experienced, disabled knitters and analyzed 20 threads from six forums that discussed accessible knitting to identify how and why disabled knitters knit, and what accessibility concerns remain. We additionally conducted an iterative design case study developing knitting tools for a knitter who found existing solutions insufficient. Our innovations improved the range of stitches she could produce. We conclude by arguing for the importance of improving tools for both pattern generation and modification as well as adaptations or modifications to existing tools such as looms to make it easier to track progress

Medical making intersects opposing value systems of a medical “do no harm” ethos and makers’ drive to innovate. Since March 2020, online maker communities have formed to design, manufacture, and distribute personal protective equipment (PPE) and other medical devices needed to fight the COVID-19 pandemic. We present a participant observation study of 14 maker communities, which have developed differing driving principles for efforts with varied access to interdisciplinary expertise on online platforms that mutually shape collective action. Over time, these communities unintentionally align towards action-oriented or regulated practices because they often lack higher level insight and agency in choosing communication platforms. In response, we recommend: regulatory bodies to build coalitions with makers, online platforms to give communities more control over the presentation of information, and repositories to balance needs to distribute information while limiting the spread of misinformation.

Unprecedented maker efforts arose in response to COVID-19 medical supply gaps worldwide. Makers in the U.S., participated in peer-production activities to manufacture personal protective equipment (PPE). Whereas, medical makers, who innovate exclusively for points of care, pivoted towards safer, reliable PPE. What were their efforts to pivot medical maker infrastructure towards reliable production of safe equipment at higher volumes? We interviewed 13 medical makers as links between institutions, maker communities, and wider regional industry networks. These medical makers organized stopgap manufacturing in institutional spaces to resolve acute shortages (March–May) and chronic shortages (May–July). They act as intermediaries in efforts to prototype and produce devices under regulatory, material, and human constraints of a pandemic. We re-frame their making efforts as repair work to offer an alternate critical view of optimism around making for crisis. We contribute an understanding of these efforts to inform infrastructure design for making with purpose and safety leading to opportunities for community production of safe devices at scale.

Tactile guiding surfaces in the built environment have held a contentious place in the process of navigation by people who are blind or visually impaired. Despite standards for tactile guiding surfaces, problems persist with inconsistent implementation, perception, and geographic orientation. We investigate the role of tactile cues in non-visual navigation and attitudes surrounding guiding surfaces through a survey of 67 people with vision impairments and ten interviews with navigation and public accessibility experts. Our participants revealed several opportunities to augment existing tactile surfaces while envisioning novel multimodal feedback solutions in immediately relevant contexts. We also propose an approach for designing and exploring low cost, multimodal tactile surfaces, which we call navtiles. Finally, we discuss practical aspects of implementation for new design alternatives such as standardization, installation, movability, discoverability, and a need for transparency. Collectively, these insights contribute to the production and implementation of novel multimodal navigation aids.

Computational handweaving combines the repeatable precision of digital fabrication with relatively high production demands of the user: a weaver must be physically engaged with the system to enact a pattern, line by line, into a fabric. Rather than approaching co-presence and repetitive labor as a negative aspect of design, we look to current practices in procedural generation (most commonly used in game design and screen-based new media art) to understand how designers can create room for suprise and emergent phenomena within systems of precision and constraint. We developed three designs for blending real-time input with predetermined pattern features. These include: using camera imagery sampled at weaving time; a 1:1 scale tool for composing patterns on the loom; and a live "Twitch'' stream where spectators determine the woven pattern. We discuss how experiential qualities of the systems led to different balances of underdetermination in procedural generation as well as how such an approach might help us think beyond an artifact/experience dichotomy in fabrication.

The Internet-of-things (IoT) embeds computing in everyday objects, but has largely focused on new devices while ignoring the home's many existing possessions. We present a field study with 10 American families to understand how these possessions could be included in the smart home through upcycling. We describe three patterns for how families collaborate around home responsibilities; we explore families' mental models of home that may be in tension with existing IoT systems; and we identify ways that families can more easily imagine a smart home that includes their existing possessions. These insights can help us design an upcycled approach to IoT that supports users in reconfiguring objects (and social roles as mediated by objects) in a way that is sensitive to what will be displaced, discarded, or made obsolete. Our findings inform the design of future lightweight systems for the upcycled home.

Automatic knitting machines are robust, digital fabrication devices that enable rapid and reliable production of attractive, functional objects by combining stitches to produce unique physical properties. However, no existing design tools support optimization for desirable physical and aesthetic knitted properties. We present KnitGIST (Generative Instantiation Synthesis Toolkit for knitting), a program synthesis pipeline and library for generating hand- and machine-knitting patterns by intuitively mapping objectives to tactics for texture design. KnitGIST generates a machine-knittable program in a domain-specific programming language.

With recent interest in shape-changing interfaces, material-driven design, wearable technologies, and soft robotics, digital fabrication of soft actuatable material is increasingly in demand. Much of this research focuses on elastomers or non-stretchy air bladders. Computationally-controlled machine knitting offers an alternative fabrication technology which can rapidly produce soft textile objects that have a very different character: breathable, lightweight, and pleasant to the touch. These machines are well established and optimized for the mass production of garments, but compared to other digital fabrication techniques such as CNC machining or 3D printing, they have received much less attention as general purpose fabrication devices. In this work, we explore new ways to employ machine knitting for the creation of actuated soft objects. We describe the basic operation of this type of machine, then show new techniques for knitting tendon-based actuation into objects. We explore a series of design strategies for integrating tendons with shaping and anisotropic texture design. Finally, we investigate different knit material properties, including considerations for motor control and sensing.

Consumer-fabrication technologies potentially improve the effectiveness and adoption of assistive technology (AT) by engaging AT users in AT creation. However, little is known about the role of clinicians in this revolution. We investigate clinical AT fabrication by working as expert fabricators for clinicians over a four-month period. We observed and co-designed AT with four occupational therapists at two clinics: a free clinic for uninsured clients, and a Veteran's Affairs Hospital. We find that existing fabrication processes, particularly with respect to rapid prototyping, do not align with clinical practice and itsdo-no-harm ethos. We recommend software solutions that would integrate into client care by: amplifying clinicians' expertise, revealing appropriate fabrication opportunities, and supporting adaptable fabrication.

We present a new type of 3D printer that combines rigid plastic printing with melt electrospinning? a technique that uses electrostatic forces to create thin fibers from a molten polymer. Our printer enables custom-shaped textile sheets (similar in feel to wool felt) to be produced alongside rigid plastic using a single material (i.e., PLA) in a single process. We contribute open-source firmware, hardware specifications, and printing parameters to achieve melt electrospinning. Our approach offers new opportunities for fabricating interactive objects and sensors that blend the flexibility, absorbency and softness of produced electrospun textiles with the structure and rigidity of hard plastic for actuation, sensing, and tactile experiences.

3D printing offers significant potential in creating highly customized interactive and functional objects. However, at present ability to manufacture functional objects is limited by available materials (e.g., various polymers) and their process properties. For instance, many functional objects need stronger materials which may be satisfied with metal printers. However, to create wholly interactive devices, we need both conductors and insulators to create wiring, and electronic components to complete circuits. Unfortunately, the single material nature of metal printing, and its inherent high temperatures, preclude this. Thus, in 3D printed devices, we have had a choice of strong materials, or embedded interactivity, but not both. In this paper, we introduce a set of techniques we call FiberWire, which leverages a new commercially available capability to 3D print carbon fiber composite objects. These objects are light weight and mechanically strong, and our techniques demonstrate a means to embed circuitry for interactive devices within them. With FiberWire, we describe a fabrication pipeline takes advantage of laser etching and fiber printing between layers of carbon-fiber composite to form low resistance conductors, thereby enabling the fabrication of electronics directly embedded into mechanically strong objects. Utilizing the fabrication pipeline, we show a range of sensor designs, their performance characterization on these new materials and finally three fully printed example object that are both interactive and mechanically strong -- a bicycle handle bar with interactive controls, a swing and impact sensing golf club and an interactive game controller (Figure 1).

Knitting creates complex, soft fabrics with unique texture properties that can be used to create interactive objects.However, little work addresses the challenges of designing and using knitted textures computationally. We present KnitPick: a pipeline for interpreting hand-knitting texture patterns into KnitGraphs which can be output to machine and hand-knitting instructions. Using KnitPick, we contribute a measured and photographed data set of 472 knitted textures. Based on findings from this data set, we contribute two algorithms for manipulating KnitGraphs. KnitCarving shapes a graph while respecting a texture, and KnitPatching combines graphs with disparate textures while maintaining a consistent shape. KnitPick is the first system to bridge the gap between hand- and machine-knitting when creating complex knitted textures.

The internet-of-things (IoT) carries substantial costs by urging households to replace their possessions with new, internet connected versions of everyday objects. Beyond financial, these costs include waste, work to arrange and orchestrate objects to suit households, and that of acquiring new skills. Upcycling domestic objects could offer households greater discretion and control over these costs by supporting the ability to tailor IoT to the home. To understand how households might do this, we conducted a home study with 10 diverse American households over 7 days to surface the approaches families are likely to use when tailoring IoT to their existing possessions. We asked family members to enact their process using endowed sticker props---IoT Stickers---to modify objects in their home. We develop a framework of how families make light weight modifications of domestic possessions, summarize trends of their object modifications, and describe the burdens such a system could impose.

In our everyday life, we interact with and benefit from objects with a wide range of material properties. In contrast, personal fabrication machines (e.g., desktop 3D printers) currently only support a much smaller set of materials. Our goal is to close the gap between current limitations and the future of multi-material printing by enabling people to explore the reuse of material from everyday objects into their custom designs. To achieve this, we develop a library of embeddables--everyday objects that can be cut, worked and embedded into 3D printable designs. We describe a design space that characterizes the geometric and material properties of embeddables. We then develop Medley---a design tool whereby users can import a 3D model, search for embeddables with desired material properties, and interactively edit and integrate their geometry to fit into the original design. Medley also supports the final fabrication and embedding process, including instructions for carving or cutting the objects, and generating optimal paths for inserting embeddables. To validate the expressiveness of our library, we showcase numerous examples augmented by embeddables that go beyond the objects' original printed materials.

Low-cost fabrication machines (e.g., 3D printers) offer the promise of creating custom-designed objects by a range of users. To maximize performance, generative design methods such as topology optimization can automatically optimize properties of a design based on high-level specifications. Though promising, such methods require people to map their design ideas--often unintuitively--to a small number of mathematical input parameters, and the relationship between those parameters and a generated design is often unclear, making it difficult to iterate a design. We present Forte, a sketch-based, real-time interactive tool for people to directly express and iterate on their designs via 2D topology optimization. Users can ask the system to add structures, provide a variation with better performance, or optimize internal material layouts. Users can globally control how much to 'deviate' from the initial sketch, or perform local suggestive editing, which interactively prompts the system to update based on the new information. Design sessions with 10 participants demonstrate that Forte empowers designers to create and explore a range of optimized designs with custom forms and styles.

Immersive experiences seek to engage the full sensory system in ways that words, pictures, or touch alone cannot. With respect to the haptic system, however, physical feedback has been provided primarily with handheld tactile experiences or vibration-based designs, largely ignoring both pressure receptors and the full upper-body area as conduits for expressing meaning that is consistent with sight and sound. We extend the potential for immersion along these dimensions with the Force Jacket, a novel array of pneumatically-actuated airbags and force sensors that provide precisely directed force and high frequency vibrations to the upper body. We describe the pneumatic hardware and force control algorithms, user studies to verify perception of airbag location and pressure magnitude, and subsequent studies to define full-torso, pressure and vibration-based feel effects such as punch, hug, and snake moving across the body. We also discuss the use of those effects in prototype virtual reality applications.

With the increasing popularity of consumer-grade 3D printing, many people are creating, and even more using, objects shared on sites such as Thingiverse. However, our formative study of 962 Thingiverse models shows a lack of re-use of models, perhaps due to the advanced skills needed for 3D modeling. An end user program perspective on 3D modeling is needed. Our framework (PARTs) empowers amateur modelers to graphically specify design intent through geometry. PARTs includes a GUI, scripting API and exemplar library of assertions which test design expectations and integrators which act on intent to create geometry. PARTs lets modelers integrate advanced, model specific functionality into designs, so that they can be re-used and extended, without programming. In two workshops, we show that PARTs helps to create 3D printable models, and modify existing models more easily than with a standard tool.

Human environments are typified by walls, homes, offices, schools, museums, hospitals and pretty much every indoor context one can imagine has walls. In many cases, they make up a majority of readily accessible indoor surface area, and yet they are static their primary function is to be a wall, separating spaces and hiding infrastructure. We present Wall++, a low-cost sensing approach that allows walls to become a smart infrastructure. Instead of merely separating spaces, walls can now enhance rooms with sensing and interactivity. Our wall treatment and sensing hardware can track users' touch and gestures, as well as estimate body pose if they are close. By capturing airborne electromagnetic noise, we can also detect what appliances are active and where they are located. Through a series of evaluations, we demonstrate Wall++ can enable robust room-scale interactive and context-aware applications.

The growing accessibility of 3D printing to everyday users has led to the rapid adoption, sharing of 3D models on sites such as Thingiverse.com, and visions of a future in which customization is a norm and 3D printing can solve a variety of real-world problems. However, in practice, creating models is difficult and many end users simply print models created by others. In this paper, we explore a specific area of model design that is a challenge for end users' measurement. When a model must conform to a specific real world goal once printed, it is important that that goal is precisely specified. We demonstrate that measurement errors are a significant (yet often overlooked) challenge for end users through a systematic study of the sources and types of measurement errors. We argue for a new design principle--accommodating measurement error--that designers, as well as novice modelers, should to use at design time. We offer two strategies--buffer insertion and replacement of minimal parts--to help designers, as well as novice modelers, to build models that are robust to measurement error. We argue that these strategies can reduce the need for and costs of iteration and demonstrate their use in a series of printed objects.

Common appliances have shifted toward flat interface panels, making them inaccessible to blind people. Although blind people can label appliances with Braille stickers, doing so generally requires sighted assistance to identify the original functions and apply the labels. We introduce Facade - a crowdsourced fabrication pipeline to help blind people independently make physical interfaces accessible by adding a 3D printed augmentation of tactile buttons overlaying the original panel. Facade users capture a photo of the appliance with a readily available fiducial marker (a dollar bill) for recovering size information. This image is sent to multiple crowd workers, who work in parallel to quickly label and describe elements of the interface. Facade then generates a 3D model for a layer of tactile and pressable buttons that fits over the original controls. Finally, a home 3D printer or commercial service fabricates the layer, which is then aligned and attached to the interface by the blind person. We demonstrate the viability of Facade in a study with 11 blind participants.

Textiles are an old and well developed technology that have many desirable characteristics. They can be easily folded, twisted, deformed, or cut; some can be stretched; many are soft. Textiles can maintain their shape when placed under tension and can even be engineered with variable stretching ability. Conversely, 3D printing is a relatively new technology that can precisely produce functional, rigid objects with custom geometry. Combining 3D printing and textiles opens up new opportunities for rapidly creating rigid objects with embedded flexibility as well as soft materials imbued with additional functionality. In this paper, we introduce a suite of techniques for integrating 3D printing with textiles during the printing process, opening up a new design space that takes inspiration from both fields. We demonstrate how the malleability, stretchability and aesthetic qualities of textiles can enhance rigid printed objects, and how textiles can be augmented with functional properties enabled by 3D printing.

We conducted a 2x2 between-subjects experiment to examine the effects of two orientation and two gaze behaviors during group conversations for a mobile, low degree-of-freedom robot. For this experiment, we designed a novel protocol to induce changes in the robot's group and study different social contexts. In addition, we implemented a perception system to track participants and control the robot's orientation and gaze with little human intervention. The results showed that the gaze behaviors under consideration affected the participants' perception of the robot's motion and that its motion affected human perception of its gaze. This mutual dependency implies that robot gaze and body motion must be designed and controlled jointly, rather than independently of each other. Moreover, the orientation behaviors that we studied led to similar feelings of inclusion and sense of belonging to the robot's group, suggesting that both can be primitives for more complex orientation behaviors.

Systems for providing mixed physical-virtual interaction on desktop surfaces have been proposed for decades, though no such systems have achieved widespread use. One major factor contributing to this lack of acceptance may be that these systems are not designed for the variety and complexity of actual work surfaces, which are often in flux and cluttered with physical objects. In this paper, we use an elicitation study and interviews to synthesize a list of ten interactive behaviors that desk-bound, digital interfaces should implement to support responsive cohabitation with physical objects. As a proof of concept, we implemented these interactive behaviors in a working augmented desk system, demonstrating their imminent feasibility.

Digital keypads have proliferated on common appliances, from microwaves and refrigerators to printers and remote controls. For blind people, such interfaces are inaccessible. We conducted a formative study with 6 blind people which demonstrated a need for custom designs for tactile labels without dependence on sighted assistance. To address this need, we introduce Facade - a crowdsourced fabrication pipeline to make physical interfaces accessible by adding a 3D printed augmentation of tactile buttons overlaying the original panel. Blind users capture a photo of an inaccessible interface with a standard marker for absolute measurements using perspective transformation. Then this image is sent to multiple crowd workers, who work in parallel to quickly label and describe elements of the interface. These labels are then used to generate 3D models for a layer of tactile and pressable buttons that fits over the original controls. Users can customize the shape and labels of the buttons using a web interface. Finally, a consumer-grade 3D printer fabricates the layer, which is then attached to the interface using adhesives. Such fabricated overlay is an inexpensive ($10) and more general solution to making physical interfaces accessible.

In this experience report, we describe the experiences of volunteer assistive device designers, clinicians, and human computer interaction and fabrication researchers who met at a summit on Do-It-Yourself Assistive Technology. From the perspectives of these stakeholders, we elucidate significant challenges of introducing rapid prototyping to the design of professional assistive technology, and opportunities for advancing assistive technology. We describe these challenges and opportunities in the context of an emerging gap between clinical and volunteer assistive device design. Whereas clinical process is fully led by the question, "will this do harm", while volunteers chaotically pursue the lofty goal of providing assistive technology to all. While all stakeholders hold the same core goals, there are many practical limitations to collaboration and development.

This paper presents a case study of three participants with upper-limb amputations working with researchers to design prosthetic devices for specific tasks: playing the cello, operating a hand-cycle, and using a table knife. Our goal was to identify requirements for a design process that can engage the assistive technology user in rapidly prototyping assistive devices that fill needs not easily met by traditional assistive technology. Our study made use of 3D printing and other playful and practical prototyping materials. We discuss materials that support on-the-spot design and iteration, dimensions along which in-person iteration is most important (such as length and angle) and the value of a supportive social network for users who prototype their own assistive technology. From these findings we argue for the importance of extensions in supporting modularity, community engagement, and relatable prototyping materials in the iterative design of prosthetics.

We describe techniques that allow inexpensive, ultra-thin, battery-free Radio Frequency Identification (RFID) tags to be turned into simple paper input devices. We use sensing and signal processing techniques that determine how a tag is being manipulated by the user via an RFID reader and show how tags may be enhanced with a simple set of conductive traces that can be printed on paper, stencil-traced, or even hand-drawn. These traces modify the behavior of contiguous tags to serve as input devices. Our techniques provide the capability to use off-the-shelf RFID tags to sense touch, cover, overlap of tags by conductive or dielectric (insulating) materials, and tag movement trajectories. Paper prototypes can be made functional in seconds. Due to the rapid deployability and low cost of the tags used, we can create a new class of interactive paper devices that are drawn on demand for simple tasks. These capabilities allow new interactive possibilities for pop-up books and other papercraft objects.

RFID tags can be used to add inexpensive, wireless, batteryless sensing to objects. However, quickly and accurately estimating the state of an RFID tag is difficult. In this work, we show how to achieve low-latency manipulation and movement sensing with off-the-shelf RFID tags and readers. Our approach couples a probabilistic filtering layer with a monte-carlo-sampling-based interaction layer, preserving uncertainty in tag reads until they can be resolved in the context of interactions. This allows designers' code to reason about inputs at a high level. We demonstrate the effectiveness of our approach with a number of interactive objects, along with a library of components that can be combined to make new designs.

We explore online reinforcement learning techniques to find good policies to control the orientation of a mobile robot during social group conversations. In this scenario, we assume that the correct behavior for the robot should convey attentiveness to the focus of attention of the conversation. Thus, the robot should turn towards the speaker. Our results from tests in a simulated environment show that a new state representation that we designed for this problem can be used to find good policies for the robot. These policies can generalize across interactions with different numbers of people and can handle various levels of sensing noise.

Several generations of inexpensive depth cameras have opened the possibility for new kinds of interaction on everyday surfaces. A number of research systems have demonstrated that depth cameras, combined with projectors for output, can turn nearly any reasonably flat surface into a touch-sensitive display. However, even with the latest generation of depth cameras, it has been difficult to obtain sufficient sensing fidelity across a table-sized surface to get much beyond a proof-of-concept demonstration. In this paper we present DIRECT, a novel touch-tracking algorithm that merges depth and infrared imagery captured by a commodity sensor. This yields significantly better touch tracking than from depth data alone, as well as any prior system. Further extending prior work, DIRECT supports arbitrary user orientation and requires no prior calibration or background capture. We describe the implementation of our system and quantify its accuracy through a comparison study of previously published, depth-based touch-tracking algorithms. Results show that our technique boosts touch detection accuracy by 15% and reduces positional error by 55% compared to the next best-performing technique.

We present CapCam, a novel technique that enables smartphones (and similar devices) to establish quick, ad-hoc connections with a host touchscreen device, simply by pressing a device to the screen's surface. Pairing data, used to bootstrap a conventional wireless connection, is transmitted optically to the phone's rear camera. This approach utilizes the near-ubiquitous rear camera on smart devices, making it applicable to a wide range of devices, both new and old. CapCam also tracks phones' physical positions on the host capacitive touchscreen without any instrumentation, enabling a wide range of targeted interactions. We quantify the communication performance of our pairing approach and demonstrate data transmission rates up to four times faster than prior camera-based techniques. To demonstrate the unique capability and utility of our system, we built a series of example applications, highlighting different interaction techniques CapCam enables.

Everyday tools and objects often need to be customized for an unplanned use or adapted for specific user, such as adding a bigger pull to a zipper or a larger grip for a pen. The advent of low-cost 3D printing offers the possibility to rapidly construct a wide range of such adaptations. However, while 3D printers are now affordable enough for even home use, the tools needed to design custom adaptations normally require skills that are beyond users with limited 3D modeling experience.

We introduce a new form of low-cost 3D printer to print interactive electromechanical objects with wound in place coils. At the heart of this printer is a mechanism for depositing wire within a five degree of freedom (5DOF) fused deposition modeling (FDM) 3D printer. Copper wire can be used with this mechanism to form coils which induce magnetic fields as a current is passed through them. Soft iron wire can additionally be used to form components with high magnetic permeability which are thus able to shape and direct these magnetic fields to where they are needed. When fabricated with structural plastic elements, this allows simple but complete custom electromagnetic devices to be 3D printed. As examples, we demonstrate the fabrication of a solenoid actuator for the arm of a Lucky Cat figurine, a 6-pole motor stepper stator, a reluctance motor rotor and a Ferrofluid display. In addition, we show how printed coils which generate small currents in response to user actions can be used as input sensors in interactive devices.

Can experimenting with three-dimensional (3D) physical objects in mixed-reality environments produce better learning and enjoyment than flat-screen two-dimensional (2D) interaction? We explored this question with EarthShake: a mixed-reality game bridging physical and virtual worlds via depth-camera sensing, designed to help children learn basic physics principles. In this paper, we report on a controlled experiment with 67 children, 4--8 years old, that examines the effect of observing physical phenomena and collaboration (pairs vs. solo). A follow-up experiment with 92 children tests whether adding simple physical control, such as shaking a tablet, improves learning and enjoyment. Our results indicate that observing physical phenomena in the context of a mixed-reality game leads to significantly more learning and enjoyment compared to screen-only versions. However, there were no significant effects of adding simple physical control or having students play in pairs vs. alone. These results and our gesture analysis provide evidence that children's science learning can be enhanced through experiencing physical phenomena in a mixed-reality environment.

DIYbio (Do It Yourself Biology) is a growing movement of scientists, hobbyists, artists, and tinkerers who practice biology outside of professional settings. In this paper, we present our work with several open source DIYbio tools, including OpenPCR and Pearl Blue Transilluminator, which can be used to test DNA samples for specific sequences. We frame these platforms as things that gather heterogeneous materials and concerns, and enable new forms of knowledge transfer. Working with these hybrid systems in professional and DIY settings, we conducted a workshop where non-biologists tested food products for genetic modifications. Our findings suggest new design directions at the intersection of biology, technology, and DIY: i) DIYbio platforms as rich tools for hybrid knowledge production; and ii) open source biology as a site for public engagement with science.

We introduce Acoustruments: low-cost, passive, and power-less mechanisms, made from plastic, that can bring rich, tangible functionality to handheld devices. Through a structured exploration, we identified an expansive vocabulary of design primitives, providing building blocks for the construction of tangible interfaces utilizing smartphones' existing audio functionality. By combining design primitives, familiar physical mechanisms can all be constructed from passive elements. On top of these, we can create end-user applications with rich, tangible interactive functionalities. Our experiments show that Acoustruments can achieve 99% accuracy with minimal training, is robust to noise, and can be rapidly prototyped. Acoustruments adds a new method to the toolbox HCI practitioners and researchers can draw upon, while introducing a cheap and passive method for adding interactive controls to consumer products.

We present a new type of 3D printer that can form precise, but soft and deformable 3D objects from layers of off-the-shelf fabric. Our printer employs an approach where a sheet of fabric forms each layer of a 3D object. The printer cuts this sheet along the 2D contour of the layer using a laser cutter and then bonds it to previously printed layers using a heat sensitive adhesive. Surrounding fabric in each layer is temporarily retained to provide a removable support structure for layers printed above it. This process is repeated to build up a 3D object layer by layer. Our printer is capable of automatically feeding two separate fabric types into a single print. This allows specially cut layers of conductive fabric to be embedded in our soft prints. Using this capability we demonstrate 3D models with touch sensing capability built into a soft print in one complete printing process, and a simple LED display making use of a conductive fabric coil for wireless power reception.

Increasingly natural, sensed, and touch-based input is being integrated into devices. Along the way, both custom and more general solutions have been developed for dealing with the uncertainty that is associated with these forms of input. However, it is difficult to provide dynamic, flexible, and continuous feedback about uncertainty using traditional interactive infrastructure. Our contribution is a general architecture with the goal of providing support for continual feedback about uncertainty. Our architecture is based on prior work in modeling uncertainty using Monte Carlo sampling, and tracks multiple interfaces -- one for each plausible and differentiable sequence of input that the user may have intended. Importantly, it considers how the presentation of uncertainty can be organized and implemented in a general way. Our primary contribution is a method for reducing the number of alternative interfaces and fusing possible interfaces into a single interface that both communicates uncertainty and allows for disambiguation. We demonstrate the value of this result through a collection of 11 new and existing feedback techniques along with two applications demonstrating the use of the feedback architecture.

We explore 3D printing physical controls whose tactile response can be manipulated programmatically through pneumatic actuation. In particular, by manipulating the internal air pressure of various pneumatic elements, we can create mechanisms that require different levels of actuation force and can also change their shape. We introduce and discuss a series of example 3D printed pneumatic controls, which demonstrate the feasibility of our approach. This includes conventional controls, such as buttons, knobs and sliders, but also extends to domains such as toys and deformable interfaces. We describe the challenges that we faced and the methods that we used to overcome some of the limitations of current 3D printing technology. We conclude with example applications and thoughts on future avenues of research.

The possibility of leveraging technology to support children's learning in the real world is both appealing and technically challenging. We have been exploring factors in tangible games that may contribute to both learning and enjoyment with an eye toward technological feasibility and scalability. Previous research found that young children learned early physics principles better when interactively predicting and observing experimental comparisons on a physical earthquake table than when seeing a video of the same. Immersing children in the real world with computer vision-based feedback appears to evoke embodied cognition that enhances learning. In the current experiment, we replicated this intriguing result of the mere difference between observing the real world versus a flat-screen. Further, we explored whether a simple and scalable addition of physical control (such as shaking a tablet) would yield an increase in learning and enjoyment. Our 2x2 experiment found no evidence that adding simple forms of hands-on control enhances learning, while demonstrating a large impact of physical observation. A general implication for educational game design is that affording physical observation in the real world accompanied by interactive feedback may be more important than affording simple hands-on control on a tablet.

Eye tracking is a compelling tool for revealing people's spatial-temporal distribution of visual attention. But quality eye tracking hardware is expensive and can only be used with one person at a time. Further, webcam eye tracking systems have significant limitations on head movement and lighting conditions that result in significant data loss and inaccuracies. To address these drawbacks, we introduce a new approach that harnesses the crowd to understand allocation of visual attention. In our approach, crowdsourcing participants use mouse clicks to self-report the positions and trajectory for the following valuable eye tracking measures: first gaze, last gaze and all gazes. We validate our crowdsourcing approach with a user study, which demonstrated good accuracy when compared to a real eye tracker. We then deployed our prototype, GazeCrowd, in a crowdsourcing setting, and showed that it accurately generated gaze heatmaps and trajectory maps. Such an approach will allow designers to evaluate and refine their visual design without requiring the use of limited/expensive eye trackers.

Appropriate robot behavior in public, open spaces cannot occur without the ability to automatically detect conversational groups of free-standing people. To this end, we propose an alternating optimization procedure that estimates lower body orientations and detects groups of interacting people. The first task is achieved by tracking the direction of the lower body of the people in the scene based on their position, their head orientation, the location of objects of interest in their vicinity, and their groups. For the second task, we propose a new group detection algorithm based on F-formation detection. This method can reason about lower body orientation distributions, and generates soft group assignments for the orientation trackers. We evaluate the proposed approach on a publicly available dataset, and show that it can improve state-of-the-art detection of non-interacting people without sacrificing group detection accuracy. This is particularly useful for robots since it provides more opportunities for starting interactions and can help estimate disengagement.

One powerful aspect of 3D printing is its ability to extend, repair, or more generally modify everyday objects. However, nearly all existing work implicitly assumes that whole objects are to be printed from scratch. Designing objects as extensions or enhancements of existing ones is a laborious process in most of today's 3D authoring tools. This paper presents a framework for 3D printing to augment existing objects that covers a wide range of attachment options. We illustrate the framework through three exemplar attachment techniques -- print-over, print-to-affix and print-through, implemented in Encore, a design tool that supports a set of analysis metrics relating to viability, durability and usability that are visualized for the user to explore design options and tradeoffs. Encore also generates 3D models for production, addressing issues such as support jigs and contact geometry between the attached part and the original object. Our validation helps to illustrate the strengths and weaknesses of each technique. For example, print-over is stronger than print-to-affix with adhesives, and all the techniques' strengths are affected by surface curvature.

Recently drivers have greatly benefited from new vehicular technologies such as in-car navigation systems, but at the same time they can be easily distracted from those technologies. Consequently, creating displays that balance the communication of information with the attentional demand imposed on the driver is of increasing importance. A set of research has been conducted to minimize the driver’s attentional cost toward the navigational displays while driving. However, accurate evaluation methods to assess the effects of these displays in realistic environments are not yet available. This article introduces the backseat driving technique for high-fidelity, safe, and inexpensive evaluation of interaction with in-car displays. This technique makes use of a real vehicle driving on a real road. As a result it allows for the exploration of some types of research questions using audio, visual, and kinesthetic stimulus at least equal in fidelity to very high-end driving simulators, without requiring such a specialized and expensive facility. Further, it allows for the employment of detailed and fine-grained measures of attentional demand, which cannot be safely used with subjects who are actually driving. Although the backseat driving technique can address only some types of questions and so is not a full replacement for high-realism driving simulators, it may offer a new approach, which augments laboratory, simulator, and real driving for many studies. As a part of the work presented here, the backseat driving technique is used to evaluate a previously developed in-car navigation display—the MOVE system. The technique allowed for new questions to be asked, which were not able to be considered in previous laboratory studies, and for the use of study measures that were only previously able to be used in the laboratory due to driving safety concerns. Specifically, the display was shown to work well when real-world stimulus are used to navigate along a real route, reducing the navigation error rate nearly threefold, and up to sixfold when compared to displays providing more or less contextual information. In addition the display was shown to cut total display fixation time (which is time spent looking away from the road) almost in half in both cases.

The average person can skillfully manipulate a plethora of tools, from hammers to tweezers. However, despite this remarkable dexterity, gestures on today's touch devices are simplistic, relying primarily on the chording of fingers: one-finger pan, two-finger pinch, four-finger swipe and similar. We propose that touch gesture design be inspired by the manipulation of physical tools from the real world. In this way, we can leverage user familiarity and fluency with such tools to build a rich set of gestures for touch interaction. With only a few minutes of training on a proof-of-concept system, users were able to summon a variety of virtual tools by replicating their corresponding real-world grasps.

This paper considers the design, construction, and example use of a new type of 3D printer which fabricates three-dimensional objects from soft fibers (wool and wool blend yarn). This printer allows the substantial advantages of additive manufacturing techniques (including rapid turn-around prototyping of physical objects and support for high levels of customization and configuration) to be employed with a new class of material. This material is a form of loose felt formed when fibers from an incoming feed of yarn are entangled with the fibers in layers below it. The resulting objects recreate the geometric forms specified in the solid models which specify them, but are soft and flexible -- somewhat reminiscent in character to hand knitted materials. This extends 3D printing from typically hard and precise forms into a new set of forms which embody a different aesthetic of soft and imprecise objects, and provides a new capability for researchers to explore the use of this class of materials in interactive devices.

Vision-based interfaces, such as those made popular by the Microsoft Kinect, suffer from the Midas Touch problem: every user motion can be interpreted as an interaction. In response, we developed an algorithm that combines facial features, body pose and motion to approximate a user's intention to interact with the system. We show how this can be used to determine when to pay attention to a user's actions and when to ignore them. To demonstrate the value of our approach, we present results from a 30-person lab study conducted to compare four engagement algorithms in single and multi-user scenarios. We found that combining intention to interact with a 'raise an open hand in front of you' gesture yielded the best results. The latter approach offers a 12% improvement in accuracy and a 20% reduction in time to engage over a baseline 'wave to engage' gesture currently used on the Xbox 360.

Tablet computers are often called upon to emulate classical pen-and-paper input. However, touchscreens typically lack the means to distinguish between legitimate stylus and finger touches and touches with the palm or other parts of the hand. This forces users to rest their palms elsewhere or hover above the screen, resulting in ergonomic and usability problems. We present a probabilistic touch filtering approach that uses the temporal evolution of touch contacts to reject palms. Our system improves upon previous approaches, reducing accidental palm inputs to 0.016 per pen stroke, while correctly passing 98% of stylus inputs.

In this study, we explored the impact of a co-located sidekick on child-robot interaction. We examined child behaviors while interacting with an expressive furniture robot and his robot lamp sidekick. The results showed that the presence of a sidekick did not alter child proximity, but did increase attention to spoken elements of the interaction. This suggests the addition of a co-located sidekick has potential to increase engagement but may not alter subtle physical interactions associated with personal space and group spatial arrangements. The findings also reinforce existing research by the community on proxemics and anthropomorphism.

The space around the body provides a large interaction volume that can allow for big interactions on small mobile devices. However, interaction techniques making use of this opportunity are underexplored, primarily focusing on distributing information in the space around the body. We demonstrate three types of around-body interaction including canvas, modal and context-aware interactions in six demonstration applications. We also present a sensing solution using standard smartphone hardware: a phone's front camera, accelerometer and inertia measurement units. Our solution allows a person to interact with a mobile device by holding and positioning it between a normal field of view and its vicinity around the body. By leveraging a user's proprioceptive sense, around-body Interaction opens a new input channel that enhances conventional interaction on a mobile device without requiring additional hardware.

The simple fact that human fingers are large and mobile devices are small has led to the perennial issue of limited surface area for touch-based interactive tasks. In response, we have developed Toffee, a sensing approach that extends touch interaction beyond the small confines of a mobile device and onto ad hoc adjacent surfaces, most notably tabletops. This is achieved using a novel application of acoustic time differences of arrival (TDOA) correlation. Previous time-of-arrival based systems have required semi-permanent instrumentation of the surface and were too large for use in mobile devices. Our approach requires only a hard tabletop and gravity -- the latter acoustically couples mobile devices to surfaces. We conducted an evaluation, which shows that Toffee can accurately resolve the bearings of touch events (mean error of 4.3° with a laptop prototype). This enables radial interactions in an area many times larger than a mobile device; for example, virtual buttons that lie above, below and to the left and right.

We present an ultra low-cost sensing system, which enables participants to see and reflect on the particulates in their air. Drawing on prior work in paper computing, we introduce small sensors for particulate pollution that can be easily assembled from common paper materials for less than $1 USD, and mailed by regular postal service to residents of entire neighborhoods, cities, or geographic regions. Recipients collect particulate samples using these sensors and mail them back to a central location, where the particles are viewed and analyzed via a microscope. The data, which includes rich images of actual air pollution particles, can then be broadcast to larger audiences. This paper details the design of our system and its deployment with a local air quality activist community. We conclude by highlighting the tradeoffs between high-tech and low-tech sensing, and suggest opportunities for tangible interaction to support rich, new ways of seeing our environment.

We present Air+Touch, a new class of interactions that interweave touch events with in-air gestures, offering a unified input modality with expressiveness greater than each input modality alone. We demonstrate how air and touch are highly complementary: touch is used to designate targets and segment in-air gestures, while in-air gestures add expressivity to touch events. For example, a user can draw a circle in the air and tap to trigger a context menu, do a finger 'high jump' between two touches to select a region of text, or drag and in-air 'pigtail' to copy text to the clipboard. Through an observational study, we devised a basic taxonomy of Air+Touch interactions, based on whether the in-air component occurs before, between or after touches. To illustrate the potential of our approach, we built four applications that showcase seven exemplar Air+Touch interactions we created.

Smartwatches are a promising new interactive platform, but their small size makes even basic actions cumbersome. Hence, there is a great need for approaches that expand the interactive envelope around smartwatches, allowing human input to escape the small physical confines of the device. We propose using tiny projectors integrated into the smartwatch to render icons on the user's skin. These icons can be made touch sensitive, significantly expanding the interactive region without increasing device size. Through a series of experiments, we show that these 'skin buttons' can have high touch accuracy and recognizability, while being low cost and power-efficient.

We present the design and deployment of a bio-electronic sensing system. This system visualizes bacterial activity inside Winogradsky columns, which incubate soil samples to culture the naturally occurring microorganisms as they process metals and nutrients in the soil. Our month-long deployments with two urban communities offer insights into individual and collective appropriations of living sensing systems. These findings reveal future design trajectories that build on emergent themes in HumanComputer Interaction (HCI), including: new perspectives on materiality, which arise from integrating organic materials with the digital; a reframing of time, as systems shift from providing instant feedback to supporting prolonged engagement; and an emphasis on collective modes of participation beyond individual behavior change.

Instant access to computing, when and where we need it, has long been one of the aims of research areas such as ubiquitous computing. In this paper, we describe the WorldKit system, which makes use of a paired depth camera and projector to make ordinary surfaces instantly interactive. Using this system, touch-based interactivity can, without prior calibration, be placed on nearly any unmodified surface literally with a wave of the hand, as can other new forms of sensed interaction. From a user perspective, such interfaces are easy enough to instantiate that they could, if desired, be recreated or modified "each time we sat down" by "painting" them next to us. From the programmer's perspective, our system encapsulates these capabilities in a simple set of abstractions that make the creation of interfaces quick and easy. Further, it is extensible to new, custom interactors in a way that closely mimics conventional 2D graphical user interfaces, hiding much of the complexity of working in this new domain. We detail the hardware and software implementation of our system, and several example applications built using the library.

We present a technology for designing curved display surfaces that can both display information and sense two dimensions of human touch. It is based on 3D printed optics, where the surface of the display is constructed as a bundle of printed light pipes, that direct images from an arbitrary planar image source to the surface of the display. This effectively decouples the display surface and image source, allowing to iterate the design of displays without requiring changes to the complex electronics and optics of the device. In addition, the same optical elements also direct light from the surface of the display back to the image sensor allowing for touch input and proximity detection of a hand relative to the display surface. The resulting technology is effective in designing compact, efficient displays of a small size; this has been applied in the design of interactive animated eyes.

We present Lumitrack, a novel motion tracking technology that uses projected structured patterns and linear optical sensors. Each sensor unit is capable of recovering 2D location within the projection area, while multiple sensors can be combined for up to six degree of freedom (DOF) tracking. Our structured light approach is based on special patterns, called m-sequences, in which any consecutive sub-sequence of m bits is unique. Lumitrack can utilize both digital and static projectors, as well as scalable embedded sensing configurations. The resulting system enables high-speed, high precision, and low-cost motion tracking for a wide range of interactive applications. We detail the hardware, operation, and performance characteristics of our approach, as well as a series of example applications that highlight its immediate feasibility and utility.

Touch input is constrained, typically only providing finger X/Y coordinates. To access and switch between different functions, valuable screen real estate must be allocated to buttons and menus, or users must perform special actions, such as touch-and-hold, double tap, or multi-finger chords. Even still, this only adds a few bits of additional information, leaving touch interaction unwieldy for many tasks. In this work, we suggest using a largely unutilized touch input dimension: shear (force tangential to a screen's surface). Similar to pressure, shear can be used in concert with conventional finger positional input. However, unlike pressure, shear provides a rich, analog 2D input space, which has many powerful uses. We put forward five classes of advanced interaction that considerably expands the envelope of interaction possible on touchscreens.

Small point lights (e.g., LEDs) are used as indicators in a wide variety of devices today, from digital watches and toasters, to washing machines and desktop computers. Although exceedingly simple in their output - varying light intensity over time - their design space can be rich. Unfortunately, a survey of contemporary uses revealed that the vocabulary of lighting expression in popular use today is small, fairly unimaginative, and generally ambiguous in meaning. In this paper, we work through a structured design process that points the way towards a much richer set of expressive forms and more effective communication for this very simple medium. In this process, we make use of five different data gathering and evaluation components to leverage the knowledge, opinions and expertise of people outside our team. Our work starts by considering what information is typically conveyed in this medium. We go on to consider potential expressive forms -- how information might be conveyed. We iteratively refine and expand these sets, concluding with ideas gathered from a panel of designers. Our final step was to make use of thousands of human judgments, gathered in a crowd-sourced fashion (265 participants), to measure the suitability of different expressive forms for conveying different information content. This results in a set of recommended light behaviors that mobile devices, such as smartphones, could readily employ.

Recent technological advances in input sensing, as well as ultra-small projectors, have opened up new opportunities for interaction -- the use of the body itself as both an input and output platform. Such on-body interfaces offer new interactive possibilities, and the promise of access to computation, communication and information literally in the palm of our hands. The unique context of on-body interaction allows us to take advantage of extra dimensions of input our bodies naturally afford us. In this paper, we consider how the arms and hands can be used to enhance on-body interactions, which is typically finger input centric. To explore this opportunity, we developed Armura, a novel interactive on-body system, supporting both input and graphical output. Using this platform as a vehicle for exploration, we proto-typed many applications and interactions. This helped to confirm chief use modalities, identify fruitful interaction approaches, and in general, better understand how interfaces operate on the body. We highlight the most compelling techniques we uncovered. Further, this paper is the first to consider and prototype how conventional interaction issues, such as cursor control and clutching, apply to the on-body domain. Finally, we bring to light several new and unique interaction techniques.

We present acoustic barcodes, structured patterns of physical notches that, when swiped with e.g., a fingernail, produce a complex sound that can be resolved to a binary ID. A single, inexpensive contact microphone attached to a surface or object is used to capture the waveform. We present our method for decoding sounds into IDs, which handles variations in swipe velocity and other factors. Acoustic barcodes could be used for information retrieval or to triggering interactive functions. They are passive, durable and inexpensive to produce. Further, they can be applied to a wide range of materials and objects, including plastic, wood, glass and stone. We conclude with several example applications that highlight the utility of our approach, and a user study that explores its feasibility.

We present an approach to 3D printing custom optical elements for interactive devices labelled Printed Optics. Printed Optics enable sensing, display, and illumination elements to be directly embedded in the casing or mechanical structure of an interactive device. Using these elements, unique display surfaces, novel illumination techniques, custom optical sensors, and embedded optoelectronic components can be digitally fabricated for rapid, high fidelity, highly customized interactive devices. Printed Optics is part of our long term vision for interactive devices that are 3D printed in their entirety. In this paper we explore the possibilities for this vision afforded by fabrication of custom optical elements using today's 3D printing technology.

Icons in graphical user interfaces convey information in a mostly universal fashion that allows users to immediately interact with new applications, systems and devices. In this paper, we define Kineticons - an iconographic scheme based on motion. By motion, we mean geometric manipulations applied to a graphical element over time (e.g., scale, rotation, deformation). In contrast to static graphical icons and icons with animated graphics, kineticons do not alter the visual content or "pixel-space" of an element. Although kineticons are not new - indeed, they are seen in several popular systems - we formalize their scope and utility. One powerful quality is their ability to be applied to GUI elements of varying size and shape from a something as small as a close button, to something as large as dialog box or even the entire desktop. This allows a suite of system-wide kinetic behaviors to be reused for a variety of uses. Part of our contribution is an initial kineticon vocabulary, which we evaluated in a 200 participant study. We conclude with discussion of our results and design recommendations.

This study explores whether certain contribution incentives for online user-generated content can undermine or enhance contributor's credibility. In an online experiment, we found that contributors who are rewarded with donations made in their names are perceived to be more credible than contributors who are financially compensated through revenue-sharing or contribute voluntarily. In addition, disclosing the chosen charity for donation can also impact credibility. Content viewer's self-identification with charity and the congruency between charity and content topic are both factors that may enhance credibility. Our findings lead to practical implications on when and how to use contribution incentives to enhance credibility.

We present SurfaceMouse, a virtual mouse for multi-touch surface computing. Although moving away from the direct touch manipulation paradigm, our system brings many sig-nificant benefits seen in absolute clutched devices to sur-face computing. Features include high and variable control device gains, several degrees of freedom in a single hand gesture, ability to target small GUI items, and a familiar method for reaching far areas of large displays. Importantly, this benefit is realized by leveraging what users already know and have tremendous experience with - physical mice. Results from our proof-of-concept evaluation reflect this; users were able to use and recognize our system with-out training or prompts. Being entirely virtual, Surface-Mouse can be implemented in existing systems with little more than a software update.

We present Pediluma, a shoe accessory that tracks and visualizes the wearer's physical activity by varying the intensity of a lighted enclosure. In particular, the more physically active the wearer is, the more the device glows. We hoped the desire to maintain a positive, "glowing" state would encourage users to engage in more physical activity. We describe our two-week, four-condition, 18-participant deployment and user study. Results indicate participants wearing our device were more physically active than participants in our three control groups (each isolating different design and experimental factors). We share the many lessons we learned from our iterative design, post-deployment data analysis and interviewers with participants.

Most LCD screens exhibit color distortions when viewed at oblique angles. Engineers have invested significant time and resources to alleviate this effect. However, the massive manufacturing base, as well as millions of in-the-wild monitors, means this effect will be common for many years to come. We take an opposite stance, embracing these optical peculiarities, and consider how they can be used in productive ways. This paper discusses how a special palette of colors can yield visual elements that are invisible when viewed straight-on, but visible at oblique angles. In essence, this allows conventional, unmodified LCD screens to output two images simultaneously - a feature normally only available in far more complex setups. We enumerate several applications that could take advantage of this ability.

We present TapSense, an enhancement to touch interaction that allows conventional surfaces to identify the type of object being used for input. This is achieved by segmenting and classifying sounds resulting from an object's impact. For example, the diverse anatomy of a human finger allows different parts to be recognized including the tip, pad, nail and knuckle - without having to instrument the user. This opens several new and powerful interaction opportunities for touch input, especially in mobile devices, where input is extremely constrained. Our system can also identify different sets of passive tools. We conclude with a comprehensive investigation of classification accuracy and training implications. Results show our proof-of-concept system can support sets with four input types at around 95% accuracy. Small, but useful input sets of two (e.g., pen and finger discrimination) can operate in excess of 99% accuracy.

Current input handling systems provide effective techniques for modeling, tracking, interpreting, and acting on user input. However, new interaction technologies violate the standard assumption that input is certain. Touch, speech recognition, gestural input, and sensors for context often produce uncertain estimates of user inputs. Current systems tend to remove uncertainty early on. However, information available in the user interface and application can help to resolve uncertainty more appropriately for the end user. This paper presents a set of techniques for tracking the state of interactive objects in the presence of uncertain inputs. These techniques use a Monte Carlo approach to maintain a probabilistically accurate description of the user interface that can be used to make informed choices about actions. Samples are used to approximate the distribution of possible inputs, possible interactor states that result from inputs, and possible actions (callbacks and feedback) interactors may execute. Because each sample is certain, the developer can specify most of the behavior of interactors in a familiar, non-probabilistic fashion. This approach retains all the advantages of maintaining information about uncertainty while minimizing the need for the developer to work in probabilistic terms. We present a working implementation of our framework and illustrate the power of these techniques within a paint program that includes three different kinds of uncertain input.

We introduce SideBySide, a system designed for ad-hoc multi-user interaction with handheld projectors. SideBySide uses device-mounted cameras and hybrid visible/infrared light projectors to track multiple independent projected images in relation to one another. This is accomplished by projecting invisible fiducial markers in the near-infrared spectrum. Our system is completely self-contained and can be deployed as a handheld device without instrumentation of the environment. We present the design and implementation of our system including a hybrid handheld projector to project visible and infrared light, and techniques for tracking projected fiducial markers that move and overlap. We introduce a range of example applications that demonstrate the applicability of our system to real-world scenarios such as mobile content exchange, gaming, and education.

Although network bandwidth has increased dramatically, high-resolution images often take several seconds to load, and considerably longer on mobile devices over wireless connections. Progressive image loading techniques allow for some visual content to be displayed prior to the whole file being downloaded. In this note, we present an empirical evaluation of popular progressive image loading methods, and derive one novel technique from our findings. Results suggest a spiral variation of bilinear interlacing can yield an improvement in content recognition time.

We present Minput, a sensing and input method that enables intuitive and accurate interaction on very small devices -- ones too small for practical touch screen use and with limited space to accommodate physical buttons. We achieve this by incorporating two, inexpensive and high-precision optical sensors (like those found in optical mice) into the underside of the device. This allows the entire device to be used as an input mechanism, instead of the screen, avoiding occlusion by fingers. In addition to x/y translation, our system also captures twisting motion, enabling many interesting interaction opportunities typically found in larger and far more complex systems.

Human perception of time is fluid, and can be manipulated in purposeful and productive ways. In this note, we propose and evaluate variations on two visual designs for progress bars that alter users' perception of time passing, and "appear" faster when in fact they are not. As a baseline, we use standard, solid-color progress bars, prevalent in many user interfaces. In a series of direct comparison tests, we are able to rank how these augmentations compare to one another. We then show that these designs yield statistically significantly shorter perceived durations than progress bars seen in many modern interfaces, including Mac OSX. Progress bars with animated ribbing that move backwards in a decelerating manner proved to have the strongest effect. In a final experiment, we measured the effect of this particular progress bar design and showed that it reduces the perceived duration among our participants by 11%.

Electronic commerce has enabled a number of online pay-for-answer services. However, despite commercial interest, we still lack a comprehensive understanding of how financial incentives support question asking and answering. Using 800 questions randomly selected from a pay-for-answer site, along with site usage statistics, we examined what factors impact askers' decisions to pay. We also explored how financial rewards affect answers, and if question pricing can help organize Q&A exchanges for archival purposes. We found that askers' decisions are two-part--whether or not to pay and how much to pay. Askers are more likely to pay when requesting facts and will pay more when questions are more difficult. On the answer side, our results support prior findings that paying more may elicit a higher number of answers and answers that are longer, but may not elicit higher quality answers (as rated by the askers). Finally, we present evidence that questions with higher rewards have higher archival value, which suggests that pricing can be used to support archival use.

A cord, although simple in form, has many interesting physical affordances that make it powerful as an input device. Not only can a length of cord be grasped in different locations, but also pulled, twisted and bent---four distinct and expressive dimensions that could potentially act in concert. Such an input mechanism could be readily integrated into headphones, backpacks, and clothing. Once grasped in the hand, a cord can be used in an eyes-free manner to control mobile devices, which often feature small screens and cramped buttons. In this note, we describe a proof-of-concept cord-based sensor, which senses three of the four input dimensions we propose. In addition to a discussion of potential uses, we also present results from our preliminary user study. The latter sought to compare the targeting performance and selection accuracy of different cord-based input modalities. We conclude with brief set of design recommendations drawn upon results from our study.

Information about the location and size of the targets that users interact with in real world settings can enable new innovations in human performance assessment and soft-ware usability analysis. Accessibility APIs provide some information about the size and location of targets. How-ever this information is incomplete because it does not sup-port all targets found in modern interfaces and the reported sizes can be inaccurate. These accessibility APIs access the size and location of targets through low-level hooks to the operating system or an application. We have developed an alternative solution for target identification that leverages visual affordances in the interface, and the visual cues produced as users interact with targets. We have used our novel target identification technique in a hybrid solution that combines machine learning, computer vision, and accessibility API data to find the size and location of targets users select with 89% accuracy. Our hybrid approach is superior to the performance of the accessibility API alone: in our dataset of 1355 targets covering 8 popular applications, only 74% of the targets were correctly identified by the API alone.

We introduce Whack Gestures, an inexact and inattentive interaction technique. This approach seeks to provide a simple means to interact with devices with minimal attention from the user -- in particular, without the use of fine motor skills or detailed visual attention (requirements found in nearly all conventional interaction techniques). For mobile devices, this could enable interaction without "getting it out," grasping, or even glancing at the device. This class of techniques is suitable for a small number of simple but common interactions that could be carried out in an extremely lightweight fashion without disrupting other activities. With Whack Gestures, users can interact by striking a device with the open palm or heel of the hand. We briefly discuss the development and use of a preliminary version of this technique and show that implementations with high accuracy and a low false positive rate are feasible.

New input technologies (such as touch), recognition based input (such as pen gestures) and next-generation interactions (such as inexact interaction) all hold the promise of more natural user interfaces. However, these techniques all create inputs with some uncertainty. Unfortunately, conventional infrastructure lacks a method for easily handling uncertainty, and as a result input produced by these technologies is often converted to conventional events as quickly as possible, leading to a stunted interactive experience. We present a framework for handling input with uncertainty in a systematic, extensible, and easy to manipulate fashion. To illustrate this framework, we present several traditional interactors which have been extended to provide feedback about uncertain inputs and to allow for the possibility that in the end that input will be judged wrong (or end up going to a different interactor). Our six demonstrations include tiny buttons that are manipulable using touch input, a text box that can handle multiple interpretations of spoken input, a scrollbar that can respond to inexactly placed input, and buttons which are easier to click for people with motor impairments. Our framework supports all of these interactions by carrying uncertainty forward all the way through selection of possible target interactors, interpretation by interactors, generation of (uncertain) candidate actions to take, and a mediation process that decides (in a lazy fashion) which actions should become final.

Physical buttons have the unique ability to provide low-attention and vision-free interactions through their intuitive tactile clues. Unfortunately, the physicality of these interfaces makes them static, limiting the number and types of user interfaces they can support. On the other hand, touch screen technologies provide the ultimate interface flexibility, but offer no inherent tactile qualities. In this paper, we describe a technique that seeks to occupy the space between these two extremes - offering some of the flexibility of touch screens, while retaining the beneficial tactile properties of physical interfaces.

In this paper, we consider a passive approach to tactile presentation based on changing the surface textures of objects that might naturally be handled by a user. This may allow devices and other objects to convey small amounts of information in very unobtrusive ways and with little attention demand. This paper considers several possible uses for this style of display and explores implementation issues. We conclude with results from our user study, which indicate that users can detect upwards of four textural states accurately with even simple materials.

Advances in electronics have brought the promise of wearable computers to near reality. Such systems can offer a highly personal and mobile information and communication infrastructure. Previous research has investigated where wearable computers can be located on the human body - critical for successful development and acceptance. However, for a location to be truly useful, it needs to not only be accessible for interaction, socially acceptable, comfortable and sufficiently stable for electronics, but also effective at conveying information. In this paper, we describe the results from a study that evaluated reaction time performance to visual stimuli at seven different body locations. Results indicate that there are numerous and statistically significant differences in the reaction time performance characteristics of these locations. We believe our findings can be used to inform the design and placement of future wearable computing applications and systems.

We present Abracadabra, a magnetically driven input technique that offers users wireless, unpowered, high fidelity finger input for mobile devices with very small screens. By extending the input area to many times the size of the device's screen, our approach is able to offer a high C-D gain, enabling fine motor control. Additionally, screen occlusion can be reduced by moving interaction off of the display and into unused space around the device. We discuss several example applications as a proof of concept. Finally, results from our user study indicate radial targets as small as 16 degrees can achieve greater than 92% selection accuracy, outperforming comparable radial, touch-based finger input.

Understanding how pointing performance varies in real world computer use and over time can provide valuable insight about how systems should accommodate changes in pointing behavior. Unfortunately, pointing data from individuals with pointing problems is rarely studied during real world use. Instead, it is most frequently evaluated in a laboratory where it is easier to collect and evaluate data. We developed a technique to collect and analyze real world pointing performance which we used to investigate the variance in performance of six individuals with a range of pointing abilities. Features of pointing performance we analyzed include metrics such as movement trajectories, clicking, and double clicking. These individuals exhibited high variance during both supervised and unsupervised (or real world) computer use across multiple login sessions. The high variance found within each participant highlights the potential inaccuracy of judging performance based on a single laboratory session.

A reliable and unobtrusive measurement of working memory load could be used to evaluate the efficacy of interfaces and to provide real-time user-state information to adaptive systems. In this paper, we describe an experiment we con-ducted to explore some of the issues around using an elec-troencephalograph (EEG) for classifying working memory load. Within this experiment, we present our classification methodology, including a novel feature selection scheme that seems to alleviate the need for complex drift modeling and artifact rejection. We demonstrate classification accuracies of up to 99% for 2 memory load levels and up to 88% for 4 levels. We also present results suggesting that we can do this with shorter windows, much less training data, and a smaller number of EEG channels, than reported previously. Finally, we show results suggesting that the models we construct transfer across variants of the task, implying some level of generality. We believe these findings extend prior work and bring us a step closer to the use of such technologies in HCI research.

In this work, we introduce the use of tags to support the near synchronous use of instant messaging. As a proof-of-concept, we developed a plug-in in Lotus Sametime, an enterprise IM client. Our plug-in supports tasks that do not need immediate attention and tasks that have deadlines. A trial deployment and survey shows that users can see the potential usefulness of such a tagging system in their IM communication. Furthermore, users rated our design intuitive and easy to use. Longer study is needed to explore communication norms that results from its use.

Responsiveness, or the time until a person responds to communication, can affect the dynamics of a conversation as well as participants' perceptions of one another. In this paper, we present a careful examination of responsiveness to instant messaging communication, showing, for example, that work-fragmentation significantly correlates with faster responsiveness. We show also that the presentation of the incoming communication significantly affects responsiveness (even more so than indicators that the communication was ongoing), suggesting the potential for dynamically influencing responsiveness. This work contributes to a better understanding of computer-mediated communication and to the design of new tools for computer-mediated communication.

There is a growing interest in applying market mechanisms to tackle everyday communication problems such as communication interruptions and communication overload. Prior analytic proofs have shown that a signaling and screening mechanism can make senders and recipients of messages better off. However, these proofs make certain assumptions that do not hold in real world environments. For example, these prior works assume that there are no transaction costs in a communication market and that monetary incentives are the only motivators in communication between strangers.

Experience sampling method (or ESM) is a common data collection method to understand user behavior and to evaluate ubiquitous computing technologies. However, ESM studies often demand too much time and commitment from participants, which leads to attrition and low compliance among participants. We introduce a new approach called experience sampling with feedback or ES+feedback that improves compliance by giving feedback to participants through various visualizations. Providing feedback to users makes the information personally relevant and increases the value of the study to participants, which increases their compliance. Our exploratory study shows that ES+feedback increases the compliance rate by 23%.

Users’ ability to accurately recall frequent, habitual activities is fundamental to a number of disciplines, from health sciences to machine learning. However, few, if any, studies exist that have assessed optimal sampling strategies for in situ self-reports. In addition, few technologies exist that facilitate benchmarking self-report accuracy for routine activities. We report on a study investigating the effect of sampling frequency of self-reports of two routine activities (sitting and walking) on recall accuracy and annoyance. We used a novel wearable sensor platform that runs a real time activity inference engine to collect in situ ground truth. Our results suggest that a sampling frequency of five to eight times per day may yield an optimal balance of recall and annoyance. Additionally, requesting self-reports at regular, predetermined times increases accuracy while minimizing perceived annoyance since it allows participants to anticipate these requests. We discuss our results and their implications for future studies.

When conversing with someone via video conference, you are provided with a virtual window into their space. However, this currently remains both flat and fixed, limiting its immersiveness. Previous research efforts have explored the use of 3D in telecommunication, and show that the additional realism can enrich the video conference experience. However, existing systems require complex sensor and cameras setups that make them infeasible for widespread adoption. We present a method for producing a pseudo-3D experience using only a single generic webcam at each end. This means nearly any computer currently able to video conference can use our technique, making it readily adoptable. Although using comparatively simple techniques, the 3D result is convincing.

Since not all persons interact with computer systems in the same way, computer systems should not interact with all individuals in the same way. This paper presents a significant step in automatically detecting characteristics of persons with a wide range of abilities based on observing their user input events. Three datasets are used to build learned statistical models on pointing data collected in a laboratory setting from individuals with varying ability to use computer pointing devices. The first dataset is used to distinguish between pointing behaviors from individuals with pointing problems vs. individuals without with 92.7% accuracy. The second is used to distinguish between pointing data from Young Adults and Adults vs. Older Adults vs. individuals with Parkinson’s Disease with 91.6% accuracy. The final data set is used to predict the need for a specific adaptation based on a user’s performance with 94.4 % accuracy. These results suggest that it may be feasible to use such models to automatically identify computer users who would benefit from accessibility tools, and to even make specific tool recommendations.

We present Scratch Input, an acoustic-based input technique that relies on the unique sound produced when a fingernail is dragged over the surface of a textured material, such as wood, fabric, or wall paint. We employ a simple sensor that can be easily coupled with existing surfaces, such as walls and tables, turning them into large, unpowered and ad hoc finger input surfaces. Our sensor is sufficiently small that it could be incorporated into a mobile device, allowing any suitable surface on which it rests to be appropriated as a gestural input surface. Several example applications were developed to demonstrate possible interactions. We conclude with a study that shows users can perform six Scratch Input gestures at about 90% accuracy with less than five minutes of training and on wide variety of surfaces.

Numerous methods have been proposed that allow mobile devices to determine where they are located (e.g., home or office) and in some cases, predict what activity the user is currently engaged in (e.g., walking, sitting, or driving). While useful, this sensing currently only tells part of a much richer story. To allow devices to act most appropriately to the situation they are in, it would also be very helpful to know about their placement - for example whether they are sitting on a desk, hidden in a drawer, placed in a pocket, or held in one's hand - as different device behaviors may be called for in each of these situations. In this paper, we describe a simple, small, and inexpensive multispectral optical sensor for identifying materials in proximity to a device. This information can be used in concert with e.g., location information, to estimate, for example, that the device is "sitting on the desk at home", or "in the pocket at work". This paper discusses several potential uses of this technology, as well as results from a two-part study, which indicates that this technique can detect placement at 94.4% accuracy with real-world placement sets.

Modern computer displays tend to be in fixed size, rigid, and rectilinear rendering them insensitive to the visual area demands of an application or the desires of the user. Foldable displays offer the ability to reshape and resize the interactive surface at our convenience and even permit us to carry a very large display surface in a small volume. In this paper, we implement four interactive foldable display designs using image projection with low-cost tracking and explore display behaviors using orientation sensitivity.

People have developed a variety of conventions for negotiating face to face interruptions. The physical distribution of teams, however, together with the use of computer mediated communication and awareness systems, fundamentally alters what information is available to a person considering an interruption of a remote collaborator. This paper presents a detailed comparison between self-reports of interruptibility, collected from participants over extended periods in their actual work environment, and estimates of this interruptibility, provided by a second set of participants based on audio and video recordings. Our results identify activities and environmental cues that affect participants' ability to correctly estimate interruptibility. We show, for example, that a closed office door had a significant effect on observers' estimation of interruptibility, but did not have an effect on participants' reports of their own interruptibility. We discuss our findings and their importance for successful design of computer-mediated communication and awareness systems.

Sensor based statistical models promise to support a variety of advances in human computer interaction, but building applications that use them is currently difficult and potential advances go unexplored. We present Subtle, a toolkit that removes some of the obstacles to developing and deploying applications using sensor based statistical models of human situations. Subtle provides an appropriate and extensible sensing library, continuous learning of personalized models, fully automated high level feature generation, and support for using learned models in deployed applications. By removing obstacles to developing and deploying sensor based statistical models, Subtle makes it easier to explore the design space surrounding sensor based statistical models of human situations. Subtle thus helps to move the focus of human computer interaction research onto applications and datasets, instead of the difficulties of developing and deploying sensor based statistical models.

If applications were able to detect a user's expertise, then software could automatically adapt to better match exper-tise. Detecting expertise is difficult because a user's skill changes as the user interacts with an application and differs across applications. This means that expertise must be sensed dynamically, continuously, and unobtrusively so as not to burden the user. We present an approach to this prob-lem that can operate without a task model based on low-level mouse and menu data which can typically be sensed across applications at the operating systems level. We have implemented and trained a classifier that can detect "nov-ice" or "skilled" use of an image editing program, the GNU Image Manipulation Program (GIMP), at 91% accuracy, and tested it against real use. In particular, we developed and tested a prototype application that gives the user dy-namic application information that differs depending on her performance.

A common task in graphical user interfaces is controlling onscreen elements using a pointer. Current adaptive pointing techniques require applications to be built using accessibility libraries that reveal information about interactive targets, and most do not handle path/menu navigation. We present a pseudo-haptic technique that is OS and application independent, and can handle both dragging and clicking. We do this by associating a small force with each past click or drag. When a user frequently clicks in the same general area (e.g., on a button), the patina of past clicks naturally creates a pseudo-haptic magnetic field with an effect similar to that ofsnapping or sticky icons. Our contribution is a bottom-up approach to make targets easier to select without requiring prior knowledge of them.

A number of projects within the computer graphics, computer vision, and human-computer interaction communities have recognized the value of using projected structured light patterns for the purposes of doing range finding, location dependent data delivery, projector adaptation, or object discovery and tracking. However, most of the work exploring these concepts has relied on visible structured light patterns resulting in a caustic visual experience. In this work, we present the first design and implementation of a high-resolution, scalable, general purpose invisible near-infrared projector that can be manufactured in a practical manner. This approach is compatible with simultaneous visible light projection and integrates well with future Digital Light Processing (DLP) projector designs -- the most common type of projectors today. By unifying both the visible and non-visible pattern projection into a single device, we can greatly simply the implementation and execution of interactive projection systems. Additionally, we can inherently provide location discovery and tracking capabilities that are unattainable using other approaches.

A problem with the location-free nature of cell phones is that callers have difficulty predicting receivers' states, leading to inappropriate calls. One promising solution involves helping callers decide when to interrupt by providing them contextual information about receivers. We tested the effectiveness of different kinds of contextual information by measuring the degree of agreement between receivers' desires and callers' decisions. In a simulation, five groups of participants played the role of ‘Callers’, choosing between making calls or leaving messages, and a sixth group played the role of ‘Receivers’, choosing between receiving calls or receiving messages. Callers were provided different contextual information about Receivers' locations, their cell phones' ringer state, the presence of others, or no information at all. Callers provided with contextual information made significantly more accurate decisions than those without it. Our results suggest that different contextual information generates different kinds of improvements: more appropriate interruptions or better avoidance of inappropriate interruptions. We discuss the results and implications for practice in the light of other important considerations, such as privacy and technological simplicity.

Text-based interpersonal communication tools such as instant messenger are widely used today. These tools often feature emoticons that people use to express emotion to some degree. However, emoticons still lack the ability to communicate the details of an emotional response, such as the speaker's tone of voice or intensity of emotion. In this paper, we hypothesize that kinetic typography - text that moves or changes over time - can address some of this problem by enhancing emotional qualities of text communication using its dynamic and expressive properties.This paper presents a study showing that a small sample of designers can create kinetic effects that end-users could employ to consistently convey emotion. In the study, three designers prepared 24 kinetic examples expressing four different emotions. We found that the examples were rated quite consistently by 66 participants. These findings provide a preliminary indication that designers can create predefined kinetic effects which can be applied to a variety of textual messages, and that these effects will reliably convey a particular emotional intent. The findings from this study inform design guidelines for designing an instant messaging client that uses kinetic typography presentation.

For the majority of us, inter-personal communication is an essential part of our daily lives. Instant Messaging, or IM, has been growing in popularity for personal and work-related communication. The low cost of sending a message, combined with the limited awareness provided by current IM systems result in messages often arriving at inconvenient or disruptive times. In a step towards solving this problem, we created statistical models that successfully predict responsiveness to incoming instant messages -- simply put: whether the receiver is likely to respond to a message within a certain time period. These models were constructed using a large corpus of real IM interaction collected from 16 participants, including over 90,000 messages. The models we present can predict, with accuracy as high as 90.1%, whether a message sent to begin a new session of communication would get a response within 30 seconds, 1, 2, 5, and 10 minutes. This type of prediction can be used, for example, to drive online-status indicators, or in services aimed at finding potential communicators.

Instant Messaging is a popular medium for both social and work-related communication. In this paper we report an investigation of the effect of interpersonal relationship on underlying basic communication characteristics (such as messaging rate and duration) using a large corpus of instant messages. Our results show that communication characteristics differ significantly for communications between users who are in a work relationship and between users who are in a social relationship. We used our findings to inform the creation of statistical models that predict the relationship between users without the use of message content -- achieving an accuracy of nearly 80% for one such model. We discuss the results of our analyses and potential uses of these models.

The home deployment of sensor-based systems offers many opportunities, particularly in the area of using sensor-based systems to support aging in place by monitoring an elder's activities of daily living. But existing approaches to home activity recognition are typically expensive, difficult to install, or intrude into the living space. This paper considers the feasibility of a new approach that "reaches into the home" via the existing infrastructure. Specifically, we deploy a small number of low-cost sensors at critical locations in a home's water distribution infrastructure. Based on water usage patterns, we can then infer activities in the home. To examine the feasibility of this approach, we deployed real sensors into a real home for six weeks. Among other findings, we show that a model built on microphone-based sensors that are placed away from systematic noise sources can identify 100% of clothes washer usage, 95% of dishwasher usage, 94% of showers, 88% of toilet flushes, 73% of bathroom sink activity lasting ten seconds or longer, and 81% of kitchen sink activity lasting ten seconds or longer. While there are clear limits to what activities can be detected when analyzing water usage, our new approach represents a sweet spot in the tradeoff between what information is collected at what cost.

Rapid, early, but rough system prototypes are becoming a standard and valued part of the user interface design process. Pen, paper, and tools like Flash™ and Director™ are well suited to creating such prototypes. However, in the case of physical forms with embedded technology, there is a lack of tools for developing rapid, early prototypes. Instead, the process tends to be fragmented into prototypes exploring forms that look like the intended product or explorations of functioning interactions that work like the intended product - bringing these aspects together into full design concepts only later in the design process. To help alleviate this problem, we present a simple tool for very rapidly creating functioning, rough physical prototypes early in the design process - supporting what amounts to interactive physical sketching. Our tool allows a designer to combine exploration of form and interactive function, using objects constructed from materials such as thumbtacks, foil, cardboard and masking tape, enhanced with a small electronic sensor board. By means of a simple and fluid tool for delivering events to "screen clippings," these physical sketches can then be easily connected to any existing (or new) program running on a PC to provide real or Wizard of Oz supported functionality.

The computer and communication systems that office workers currently use tend to interrupt at inappropriate times or unduly demand attention because they have no way to determine when an interruption is appropriate. Sensor?based statistical models of human interruptibility offer a potential solution to this problem. Prior work to examine such models has primarily reported results related to social engagement, but it seems that task engagement is also important. Using an approach developed in our prior work on sensor?based statistical models of human interruptibility, we examine task engagement by studying programmers working on a realistic programming task. After examining many potential sensors, we implement a system to log low?level input events in a development environment. We then automatically extract features from these low?level event logs and build a statistical model of interruptibility. By correctly identifying situations in which programmers are non?interruptible and minimizing cases where the model incorrectly estimates that a programmer is non?interruptible, we can support a reduction in costly interruptions while still allowing systems to convey notifications in a timely manner.

The subArctic user interface toolkit has extensibility as one of its central goals. It seeks not only to supply a powerful library of reusable interactive objects, but also make it easy to create new, unusual, and highly customized interactions tailored to the needs of particular interfaces or task domains. A central part of this extensibility is the input model used by the toolkit. The subArctic input model provides standard reusable components that implement many typical input handling patterns for the programmer, allows inputs to be handled in very flexible ways, and allows the details of how inputs are handled to be modified to meet custom needs. This paper will consider the structure and operation of the subArctic input handling mechanism. It will demonstrate the flexibility of the system through a series of examples, illustrating techniques that it enables - many of which would be very difficult to implement in most toolkits.

In-vehicle navigation has changed substantially in recent years, due to the advent of computer generated maps and directions. However, these maps are still problematic, due to a mismatch between the complexity of the maps and the attentional demands of driving. In response to this problem, we are developing the MOVE (Maps Optimized for Vehicular Environments) system. This system will provide situationally appropriate map information by presenting information that uses appropriate amounts of the driver's attention. In this paper, we describe our findings of studies to help shape the design of the MOVE system, including studies on map reading and in-vehicle navigation, and studies on the effectiveness of a variety of contextually optimized route map visualizations in a simulated driving context.Results show that contextually optimized displays designed for the MOVE system should significantly reduce perceptual load in the context of driving. In our laboratory experiment there was a six-fold decrease in the total map display fixation time and nearly threefold decrease in the number of glances needed to interpret the contextually optimized display compared to a static display.

Video projectors have typically been used to display images on surfaces whose geometric relationship to the projector remains constant, such as walls or pre-calibrated surfaces. In this paper, we present a technique for projecting content onto moveable surfaces that adapts to the motion and location of the surface to simulate an active display. This is accomplished using a projector based location tracking techinque. We use light sensors embedded into the moveable surface and project low-perceptibility Gray-coded patterns to first discover the sensor locations, and then incrementally track them at interactive rates. We describe how to reduce the perceptibility of tracking patterns, achieve interactive tracking rates, use motion modeling to improve tracking performance, and respond to sensor occlusions. A group of tracked sensors can define quadrangles for simulating moveable displays while single sensors can be used as control inputs. By unifying the tracking and display technology into a single mechanism, we can substantially reduce the cost and complexity of implementing applications that combine motion tracking and projected imagery.

A person seeking another person's attention is normally able to quickly assess how interruptible the other person currently is. Such assessments allow behavior that we consider natural, socially appropriate, or simply polite. This is in sharp contrast to current computer and communication systems, which are largely unaware of the social situations surrounding their usage and the impact that their actions have on these situations. If systems could model human interruptibility, they could use this information to negotiate interruptions at appropriate times, thus improving human computer interaction.This article presents a series of studies that quantitatively demonstrate that simple sensors can support the construction of models that estimate human interruptibility as well as people do. These models can be constructed without using complex sensors, such as vision-based techniques, and therefore their use in everyday office environments is both practical and affordable. Although currently based on a demographically limited sample, our results indicate a substantial opportunity for future research to validate these results over larger groups of office workers. Our results also motivate the development of systems that use these models to negotiate interruptions at socially appropriate times.

Toolkits and other tools have dramatically reduced the time and technical expertise needed to design and implement graphical user interfaces (GUIs) allowing high-quality, iterative, user-centered design to become a common practice. Unfortunately the generation of functioning prototypes for physical interactive devices as not had similar support -- it still requires substantial time and effort by individuals with highly specialized skills and tools. This creates a divide between a designers' ability to explore form and interactivity of product designs and the ability to iterate on the basis of high fidelity interactive experiences with a functioning prototype. To help overcome this difficulty we have developed the Calder hardware toolkit. Calder is a development environment for rapidly exploring and prototyping functional physical interactive devices. Calder provides a set of reusable small input and output components, and integration into existing interface prototyping environments. These components communicate with a computer using wired and wireless connections. Calder is a tool targeted toward product and interaction designers to aid them in their early design process. In this paper we describe the process of gaining an understanding of the needs and workflow habits of our target users to generate a collection of requirements for such a toolkit. We describe technical challenges imposed by these needs, and the specifics of design and implementation of the toolkit to meet these challenges.

Current systems often create socially awkward interruptions or unduly demand attention because they have no way of knowing if a person is busy and should not be interrupted. Previous work has examined the feasibility of using sensors and statistical models to estimate human interruptibility in an office environment, but left open some questions about the robustness of such an approach. This paper examines several dimensions of robustness in sensor-based statistical models of human interruptibility. We show that real sensors can be constructed with sufficient accuracy to drive the predictive models. We also create statistical models for a much broader group of people than was studied in prior work. Finally, we examine the effects of training data quantity on the accuracy of these models and consider tradeoffs associated with different combinations of sensors. As a whole, our analyses demonstrate that sensor-based statistical models of human interruptibility can provide robust estimates for a variety of office workers in a range of circumstances, and can do so with accuracy as good as or better than people. Integrating these models into systems could support a variety of advances in human computer interaction and computer-mediated communication. Author Keywords Situationally appropriate interaction, managing human attention, sensor-based interfaces, context-aware computing, machine learning.

The growing use of Instant Messaging for social and work-related communication has created a situation where incoming messages often become a distraction to users while they are performing important tasks. Staying on task at the expense of responsiveness to IM buddies may portray the users as impolite or even rude. Constantly attending to IM, on the other hand, may prevent users from performing tasks efficiently, leaving them frustrated. In this paper we present a tool that augments a commercial IM client by automatically increasing the salience of incoming messages that may deserve immediate attention, helping users decide whether or not to stay on task.

Light Emitting Diodes (LEDs) offer long life, low cost, efficiency, brightness, and a full range of colors. Because of these properties, they are widely used for simple displays in electronic devices. A previously characterized, but little known property of LEDs allows them to be used as photo sensors. In this paper, we show how this capability can be used to turn unmodified, off the shelf, LED arrays into touch sensitive input devices (while still remaining capable of producing output). The technique is simple and requires little or no extra hardware - in some cases operating with the same micro-controller based circuitry normally used to produce output, requiring only software changes. We will describe a simple hybrid input/output device prototype implemented with this technique, and discuss the design opportunities that this type of device opens up.

In this paper we present a system for providing tactile feedback for stylus-based touch-screen displays. The Haptic Pen is a simple low-cost device that provides individualized tactile feedback for multiple simultaneous users and can operate on large touch screens as well as ordinary surfaces. A pressure-sensitive stylus is combined with a small solenoid to generate a wide range of tactile sensations. The physical sensations generated by the Haptic pen can be used to enhance our existing interaction with graphical user interfaces as well as to help make modern computing systems more accessible to those with visual or motor impairments.

Projection technology typically places several constraints on the geometric relationship between the projector and the projection surface to obtain an undistorted, properly sized image. In this paper we describe a simple, robust, fast, and low-cost method for automatic projector calibration that eliminates many of these constraints. We embed light sensors in the target surface, project Gray-coded binary patterns to discover the sensor locations, and then prewarp the image to accurately fit the physical features of the projection surface. This technique can be expanded to automatically stitch multiple projectors, calibrate onto non-planar surfaces for object decoration, and provide a method for simple geometry acquisition.

Recent work is beginning to reveal the potential of numerical optimization as an approach to generating interfaces and displays. Optimization-based approaches can often allow a mix of independent goals and constraints to be blended in ways that are difficult to describe algorithmically. While optimization-based techniques appear to offer several potential advantages, further research in this area is hampered by the lack of appropriate tools. Optimization toolkits do exist, but they typically require substantial specialized knowledge because they have been designed for traditional optimization problems.GADGET is an experimental toolkit to support optimization as an approach to interface and display generation. GADGET provides three core abstractions, initializers, iterations, and evaluations. An initializer creates an initial solution to be optimized, based on an existing algorithm or randomly. Iterations are responsible for transforming one potential solution into another, typically using methods that are at least partially random. Finally, evaluations are used for judging the different notions of goodness in a solution. Together with a evaluation standardization framework, support for generic properties integrated with an efficient lazy evaluation framework, and a library of reusable iterations and evaluations, the abstractions provided by GADGET simplify the development of optimization-based approaches to interface and display generation.

Kinetic (dynamic) typography has demonstrated the ability to add significant emotive content and appeal to expressive text, allowing some of the qualities normally found in film and the spoken word to be added to static text. Kinetic typography has been widely and successfully used in film title sequences as well as television and computer-based advertising. However, its communicative abilities have not been widely studied, and its potential has rarely been exploited outside these areas. This is partly due to the difficulty in creating kinetic typography with current tools, often requiring hours of work to animate a single sentence.In this paper, we present the Kinedit system, a basic authoring tool that takes initial steps toward remedying this situation and hence promoting exploration of the communicative potential of kinetic typography for personal communication. Kinedit is informed by systematic study and characterization of a corpus of examples, and iterative involvement and validation by designers throughout the development process. We describe the tool and its underlying technology, usage experiences, lessons learned, and next steps.

A person seeking someone else's attention is normally able to quickly assess how interruptible they are. This assessment allows for behavior we perceive as natural, socially appropriate, or simply polite. On the other hand, today's computer systems are almost entirely oblivious to the human world they operate in, and typically have no way to take into account the interruptibility of the user. This paper presents a Wizard of Oz study exploring whether, and how, robust sensor-based predictions of interruptibility might be constructed, which sensors might be most useful to such predictions, and how simple such sensors might be.The study simulates a range of possible sensors through human coding of audio and video recordings. Experience sampling is used to simultaneously collect randomly distributed self-reports of interruptibility. Based on these simulated sensors, we construct statistical models predicting human interruptibility and compare their predictions with the collected self-report data. The results of these models, although covering a demographically limited sample, are very promising, with the overall accuracy of several models reaching about 78%. Additionally, a model tuned to avoiding unwanted interruptions does so for 90% of its predictions, while retaining 75% overall accuracy.

Recent work is beginning to reveal the potential of numerical optimization as an approach to generating interfaces and displays. Optimization-based approaches can often allow a mix of independent goals and constraints to be blended in ways that would be difficult to describe algorithmically. While optimization-based techniques appear to offer several potential advantages, further research in this area is hampered by the lack of appropriate tools. This paper presents GADGET, an experimental toolkit to support optimization for interface and display generation. GADGET provides convenient abstractions of many optimization concepts. GADGET also provides mechanisms to help programmers quickly create optimizations, including an efficient lazy evaluation framework, a powerful and configurable optimization structure, and a library of reusable components. Together these facilities provide an appropriate tool to enable exploration of a new class of interface and display generation techniques.

In order to create and use rich custom appearances, designers are often forced to introduce an unnatural gap into the design process. For example, a designer creating a skin for a music player must separately specify the appearance of the elements in the music player skin and the mapping between these visual elements and the functionality provided by the music player. This gap between appearance and semantic meaning creates a number of problems. We present a set of techniques that allows designers to use their preferred drawing tool to specify both appearance and semantic meaning. We demonstrate our techniques in an unmodified version of Adobe Photoshop®, but our techniques are general and adaptable to nearly any layered drawing package.

Kinetic typography --- text that uses movement or other temporal change --- has recently emerged as a new form of communication. As we hope to illustrate in this paper, kinetic typography can be seen as bringing some of the expressive power of film --- such as its ability to convey emotion, portray compelling characters, and visually direct attention --- to the strong communicative properties of text. Although kinetic typography offers substantial promise for expressive communications, it has not been widely exploited outside a few limited application areas (most notably in TV advertising). One of the reasons for this has been the lack of tools directly supporting it, and the accompanying difficulty in creating dynamic text. This paper presents a first step in remedying this situation --- an extensible and robust system for animating text in a wide variety of forms. By supporting an appropriate set of carefully factored abstractions, this engine provides a relatively small set of components that can be plugged together to create a wide range of different expressions. It provides new techniques for automating effects used in traditional cartoon animation, and provides specific support for typographic manipulations.

The proliferation of wireless handheld devices is placing the World Wide Web in the palms of users, but this convenience comes at a high interactive cost. The Web that came of age on the desktop is ill-suited for use on the small displays of handhelds. Today, handheld browsing often feels like browsing on a PC with a shrunken desktop. Overreliance on scrolling is a big problem in current handheld browsing. Users confined to viewing a small portion of each page often lack a sense of the overall context --- they may feel lost in a large page and be forced to remember the locations of items as those items scroll out of view. In this paper, we present a synthesis of interaction techniques to address these problems. We implemented these techniques in a prototype, WebThumb, that can browse the live Web.

Ordinary paper offers properties of readability, fluidity, flexibility, cost, and portability that current electronic devices are often hard pressed to match. In fact, a lofty goal for many interactive systems is to be "as easy to use as pencil and paper". However, the static nature of paper does not support a number of capabilities, such as search and hyperlinking that an electronic device can provide. The Paper PDA project explores ways in which hybrid paper electronic interfaces can bring some of the capabilities of the electronic medium to interactions occurring on real paper. Key to this effort is the invention of on-paper interaction techniques which retain the flexibility and fluidity of normal pen and paper, but which are structured enough to allow robust interpretation and processing in the digital world. This paper considers the design of a class of simple printed templates that allow users to make common marks in a fluid fashion, and allow additional gestures to be invented by the users to meet their needs, but at the same time encourages marks that are quite easy to recognize.

Normally, the primary purpose of an information display is to convey information. If information displays can be aesthetically interesting, that might be an added bonus. This paper considers an experiment in reversing this imperative. It describes the Kandinsky system which is designed to create displays which are first aesthetically interesting, and then as an added bonus, able to convey information. The Kandinsky system works on the basis of aesthetic properties specified by an artist (in a visual form). It then explores a space of collages composed from information bearing images, using an optimization technique to find compositions which best maintain the properties of the artist's aesthetic expression.

Interfaces based on recognition technologies are used extensively in both the commercial and research worlds. But recognizers are still error-prone, and this results in human performance problems, brittle dialogues, and other barriers to acceptance and utility of recognition systems. Interface techniques specialized to recognition systems can help reduce the burden of recognition errors, but building these interfaces depends on knowledge about the ambiguity inherent in recognition. We have extended a user interface toolkit in order to model and to provide structured support for ambiguity at the input event level. This makes it possible to build re-usable interface components for resolving ambiguity and dealing with recognition errors. These interfaces can help to reduce the negative effects of recognition errors. By providing these components at a toolkit level, we make it easier for application writers to provide good support for error handling. Further, with this robust support, we a...

Because of its promise of natural interaction, recognition is coming into its own as a mainstream technology for use with computers. Both commercial and research applications are beginning to use it extensively. However the errors made by recognizers can be quite costly, and this is increasingly becoming a focus for researchers. We present a survey of existing error correction techniques in the user interface. These mediation techniques most commonly fall into one of two strategies, repetition and choice. Based on the needs uncovered by this survey, we have developed OOPS, a toolkit that supports resolution of input ambiguity through mediation. This paper describes four new interaction techniques built using OOPS, and the toolkit mechanisms required to build them. These interaction techniques each address problems not directly handled by standard approaches to mediation, and can all be re-used in a variety of settings. INTRODUCTION Because of its promise of natural interaction, recog...

A user interface software tool helps developers design and implement the user interface. Research on past tools has had enormous impact on today's developers—virtually all applications today are built using some form of user interface tool. In this article, we consider cases of both success and failure in past user interface tools. From these cases we extract a set of themes which can serve as lessons for future work. Using these themes, past tools can be characterized by what aspects of the user interface they addressed, their threshold and ceiling, what path of least resistance they offer, how predictable they are to use, and whether they addressed a target that became irrelevant. We believe the lessons of these past themes are particularly important now, because increasingly rapid technological changes are likely to significantly change user interfaces. We are at the dawn of an era where user interfaces are about to break out of the “desktop” box where they have been stuck for the past 15 years. The next millenium will open with an increasing diversity of user interface on an increasing diversity of computerized devices. These devices include hand-held personal digital assistants (PDAs), cell phones, pages, computerized pens, computerized notepads, and various kinds of desk and wall size-computers, as well as devices in everyday objects (such as mounted on refridgerators, or even embedded in truck tires). The increased connectivity of computers, initially evidenced by the World Wide Web, but spreading also with technologies such as personal-area networks, will also have a profound effect on the user interface to computers. Another important force will be recognition-based user interfaces, especially speech, and camera-based vision systems. Other changes we see are an increasing need for 3D and end-user customization, programming, and scripting. All of these changes will require significant support from the underlying user interface sofware tools.

This paper describes an architecture for supporting interface attuchments - small interactive programs which are designed to augment the functionality of other applications. This architecture is designed to work with a diverse set of conventional applications, but require only a minimal set of hooks into those applications. In order to achieve this, the work described here concentrates on what we will call observational attachments, a subclass of attachments that operate primarily by observing and manipulating the surface representations of applications - that is the visual information that applications would normally display on the screen or print. These attachments can be thought of as looking over the shoulder of the user to assist with various tasks. By requiring very little modification to, or help from, the applications they augment, this approach supports the creation of a set of uniform services that can be applied across a more diverse set of applications than traditional approaches.

Most current interface designs require that the user focus their attention on them in order to be of value. However, as the price of computation falls, and computational capabilities make their way into many everyday objects, the demand for attention from many different directions may begin to seriously reduce the usefulness of these computational objects. Ambient information displays are intended to fit in a part of the interface design space that does not have this property. They are designed to convey background or context information that the user may or may not wish to attend to at any given time. Ambient Displays are designed to work primarily in the periphery of a user's awareness, moving to the center of attention only when appropriate and desirable. This paper describes a new ambient information display that is designed to give a rich medium of expression placed within an aesthetically pleasing decorative object. This display — the Information Percolator — is formed by air bubbles rising up tubes of water. By properly controlling the release of air, a set of pixels which scroll up the display is created. This allows a rendition of any (small, black and white) image to be displayed. The detailed design and construction of this display device will be considered, along with several applications.

It is well known that paper is a very fluid, natural, and easy to use medium for manipulating some kinds of information. It is familiar, portable, flexible, inexpensive, and offers good readability properties. Paper also has well known limitations when compared with electronic media. Work in hybrid paper electronic interfaces seeks to bring electronic capabilities to real paper in order to obtain the best properties of each. This paper describes a hybrid paper electronic system — the Paper PDA — which is designed to allow electronic capabilities to be employed within a conventional paper notebook, calendar, or organizer. The Paper PDA is based on a simple observation: a paper notebook can be synchronized with a body of electronic information much like an electronic PDA can be synchronized with information hosted on a personal computer. This can be accomplished by scanning, recognizing and processing its contents, then printing a new copy. This paper introduces the Paper PDA concept and considers interaction techniques and applications designed to work within the Paper PDA. The StickerLink technique supports on-paper hyperlinking using removable paper stickers. Two applications are also considered which look at aspects of electronic communications via the Paper PDA.

A central goal of many user interface development tools has been to make the construction of high quality interfaces easy enough that iterative design approaches could be a practical reality. In the last 15 years significant advances in this regard have been achieved. However, the evaluation portion of the iterative design process has received relatively little support from tools. Even though advances have also been made in usability evaluation methods, nearly all evaluation is still done “by hand,” making it more expensive and difficult than it might be. This paper considers a partial implementation of the CRITIQUE usability evaluation tool that is being developed to help remedy this situation by automating a number of evaluation tasks. This paper will consider techniques used by the system to produce predictive models (keystroke level models and simplified GOMS models) from demonstrations of sample tasks in a fraction of the time needed by conventional handcrafting methods. A preliminary comparison of automatically generated models with models created by an expert modeler show them to produce very similar predictions (within 2%). Further, because they are automated, these models promise to be less subject to human error and less affected by the skill of the modeler.

Article Ubiquitous collaboration via surface representations Share on Authors: Dan R. Olsen Human Computer Interaction Institite, Carnegie Mellon University, Pittsburg, PA Human Computer Interaction Institite, Carnegie Mellon University, Pittsburg, PAView Profile , Scott E. Hudson Human Computer Interaction Institite, Carnegie Mellon University, Pittsburg, PA Human Computer Interaction Institite, Carnegie Mellon University, Pittsburg, PAView Profile , Matt Phelps Human Computer Interaction Institite, Carnegie Mellon University, Pittsburg, PA Human Computer Interaction Institite, Carnegie Mellon University, Pittsburg, PAView Profile , Jeremy Heiner Human Computer Interaction Institite, Carnegie Mellon University, Pittsburg, PA Human Computer Interaction Institite, Carnegie Mellon University, Pittsburg, PAView Profile , Thom Verratti Human Computer Interaction Institite, Carnegie Mellon University, Pittsburg, PA Human Computer Interaction Institite, Carnegie Mellon University, Pittsburg, PAView Profile Authors Info & Claims CSCW '98: Proceedings of the 1998 ACM conference on Computer supported cooperative workNovember 1998 Pages 129–138https://doi.org/10.1145/289444.289487Published:01 November 1998 10citation527DownloadsMetricsTotal Citations10Total Downloads527Last 12 Months4Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Paper is a very convenient medium for presenting information.It is familiar, flexible, portable, inexpensive, user modifiable, and offers better readability properties than existing electronic displays.However, paper displays are static and do not offer capabilities such as dynamic content, and hyperlinking that can be provided with electronic media.PaperLink is a system which augments paper documents with electronic features.PaperLink uses a highlighter pen augmented with a camera, along with simple computer vision and pattern recognition techniques, to allow a user to make marks on paper which can have associations and meaning in an electronic world, and to "pick up" printed material for use as electronic input.This paper will consider the prototype PaperLink hardware and software system, and its application to hyperlinking from paper to electronic content.

Article Free Access Share on Making computers easier for older adults to use: area cursors and sticky icons Authors: Aileen Worden Psychology, Georgia Institute of Technology, Atlanta, Georgia Psychology, Georgia Institute of Technology, Atlanta, GeorgiaView Profile , Nef Walker Psychology, Georgia Institute of Technology, Atlanta, Georgia Psychology, Georgia Institute of Technology, Atlanta, GeorgiaView Profile , Krishna Bharat GVR, Georgia Institute of Technology, Atlanta, Georgia GVR, Georgia Institute of Technology, Atlanta, GeorgiaView Profile , Scott Hudson GVR, Georgia Institute of Technology, Atlanta, Georgia GVR, Georgia Institute of Technology, Atlanta, GeorgiaView Profile Authors Info & Claims CHI '97: Proceedings of the ACM SIGCHI Conference on Human factors in computing systemsMarch 1997 Pages 266–271https://doi.org/10.1145/258549.258724Published:27 March 1997Publication History 193citation3,057DownloadsMetricsTotal Citations193Total Downloads3,057Last 12 Months119Last 6 weeks16 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

In this paper we present a simple but general set of techniques for modifying output in a 2D user interface toolkit. We use a combination of simple subclassing, wrapping, and collusion between parent and output objects to produce arbitrary sets of composable output transformations. The techniques described here allow rich output effects to be added to most, if not all, existing interactors in an application, without the knowledge of the interactors themselves. This paper explains how the approach works, discusses a number of example effects that have been built, and describes how the techniques presented here could be extended to work with other toolkits. We address issues of input by examining a number of extensions to the toolkit input subsystem to accommodate transformed graphical output. Our approach uses a set of "hooks" to undo output transformations when input is to be dispatched. KEYWORDS: User Interface Toolkits, Output, Rendering, Interactors, Drawing Effects. INTRODUCTION ...

The visual and event driven nature of modem user interfaces, while a boon to users, can also make them more difficult to debug than conventional programs.This is because only the very surface representation of interactive objects -their final visual appearance -is visible to the programmer on the screen.The remaining "programming details" of the object remain hidden.If the appearance or behavior of an object is incorrect, often few clues are visible to indicate the cause.One must usually turn to text oriented debugging techniques (debuggers or simply print statements) which are separate from the interface, and often cumbersome to use with event-driven control flow.This paper describes a new class of techniques designed to aid in the debugging of user interfaces by making more of the invisible, visible.This class of techniques: debugging lenses, makes use of transparent lens interaction techniques to show debugging information.It is designed to work in situ -in the context of a running interface, without stopping or interfering with that interface.-Thispaper describes and motivates the class of techniques, gives a number of specific examples of debugging lenses, and ,describes their implementation in' the subArctic user interface toolkit.

Article Free Access Share on Supporting dynamic downloadable appearances in an extensible user interface toolkit Authors: Scott E. Hudson Human Computer Interaction Institute, Carnegie Mellon University, 5000 Forbes Ave., Pittsburgh, PA Human Computer Interaction Institute, Carnegie Mellon University, 5000 Forbes Ave., Pittsburgh, PAView Profile , Ian Smith Graphics, Visualization and Usability Center, and College of Computing, Georgia Institute of Technology, Atlanta, Georgia Graphics, Visualization and Usability Center, and College of Computing, Georgia Institute of Technology, Atlanta, GeorgiaView Profile Authors Info & Claims UIST '97: Proceedings of the 10th annual ACM symposium on User interface software and technologyOctober 1997 Pages 159–168https://doi.org/10.1145/263407.263539Published:01 October 1997Publication History 12citation443DownloadsMetricsTotal Citations12Total Downloads443Last 12 Months29Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

This paper describes a fundamental dual tradeoff that occurs in systems supporting awareness for distributed work groups, and presents several specific new techniques which illustrate good compromise points within this tradeoff space. This dual tradeoff is between privacy and awareness, and between awareness and disturbance. Simply stated, the more information about oneself that leaves your work area, the more potential for awareness of you exists for your colleagues. Unfortunately, this also represents the greatest potential for intrusion on your privacy. Similarly, the more information that is received about the activities of colleagues, the more potential awareness we have of them. However, at the same time, the more information we receive, the greater the chance that the information will become a disturbance to our normal work. This dual tradeoff seems to be a fundamental one. However, by carefully examining awareness problems in the light of this tradeoff it is possible to devise techniques which expose new points in the design space. These new points provide different types and quantities of information so that awareness can be achieved without invading the privacy of the sender, or creating a disturbance for the receiver. This paper presents four such techniques, each based on a careful selection of the information transmitted.

Constraint systems have been used for some time to implement various components of a user interface.High level support for flexible screen layou~t has been among the more important uses; layout constraints in a user interface toolkit provide a declarative mechanism for controlling the size and position of objects in an interactive display, along with an efficient update mechanism for maintaining display layouts automatically in the face of dynamic changes.This paper describes a new technique for implementing one-way layout constraints which overcomes a substantial limitation of previous systems.In particular, it allows constraints to be implemented in an extremely small amount of space -as little as 17 bits per constraint and still maintain the level of performance needed for good interactive response.These ultralightweight constraints, while not handling all (cases, cover most relationships used for layout, and allow conventional constraints to be applied when needed.This paper will consider both a general technique for ultra-lightweight layout constraints and its specific implementation in a new JavaTM-basecl user interface toolkit.

Our work concerns visualizing the information space of hypermedia systems using multiple hierarchical views. Although overview diagrams are useful for helping the user to navigate in a hypermedia system, for any real-world system they become too complicated and large to be really useful. This is because these diagrams represent complex network structures which are very difficult to visualize and comprehend. On the other hand, effective visualizations of hierarchies have been developed. Our strategy is to provide the user with different hierarchies, each giving a different perspective to the underlying information space, to help the user better comprehend the information. We propose an algorithm based on content and structural analysis to form hierarchies from hypermedia networks. The algorithm is automatic but can be guided by the user. The multiple hierarchies can be visualized in various ways. We give examples of the implementation of the algorithm on two hypermedia systems. KEYWORD...

No abstract available.

This paper describes Doppler a new, fast algorithm for supporting concurrent, one-way constraints between objects situated in multiple address spaces. Because of their declarative nature, convenience, low amortized cost, and good match to interface tasks, constraints have been used to support a variety of user-interface activities. Unfortunately, nearly all existing constraint maintenance algorithms are sequential in nature, and cannot function effectively in a concurrent or distributed setting. The Doppler algorithm overcomes these limitations. It is a highly efficient distributed and concurrent algorithm (based on an efficient sequential algorithm for incremental, lazy updates). Doppler relies solely on asynchronous message passing, and does not require shared memory, synchronized clocks, or a global synchronization mechanism. It supports a high degree of concurrency by efficiently tracking potential cause and effect relationships between reads and writes, and allowing all causally i...

This paper describes a novel approach to authoring multimedia documents based on the walk-through paradigm. Using this approach, multimedia authoring tasks can be performed in the context of the multimedia presentation under construction. It greatly simplifies the authoring process by hiding the use of composition constructs and eliminating the turn-around time from the editing and testing parts of the development cycle. End user multimedia authoring and fast prototyping can therefore be realized.

This paper describes a new interactive environment for user interface specification which is based on an enhanced spreadsheet model of computation. This environment allows sophisticated graphical user interfaces with dynamic feedback to be implemented with little or no explicit programming. Its goal is to support user interface specification by nonprogramming experts in human factors, visual design, or the application domain. In addition, the system is designed to allow sophisticated end-users to modify and customize their own interfaces. The system is based on a data flow model of computation. This model is presented to the interface designer in the form of a spreadsheet enhanced with new constructs for easier programming and reuse. These constructs include an improved interactive programming environment, a prototype-instance-based inheritance system, support for composition, abstraction, and customization using indirect references, the addition of support for graphical inputs and outputs, and support for the encapsulation of application data structures and routines within system objects.

Article Animation support in a user interface toolkit: flexible, robust, and reusable abstractions Share on Authors: Scott E. Hudson Graphics Visualization and Usability Center, College of Computing, Georgia Institute of Technology, Atlanta, GA Graphics Visualization and Usability Center, College of Computing, Georgia Institute of Technology, Atlanta, GAView Profile , John T. Stasko Graphics Visualization and Usability Center, College of Computing, Georgia Institute of Technology, Atlanta, GA Graphics Visualization and Usability Center, College of Computing, Georgia Institute of Technology, Atlanta, GAView Profile Authors Info & Claims UIST '93: Proceedings of the 6th annual ACM symposium on User interface software and technologyDecember 1993 Pages 57–67https://doi.org/10.1145/168642.168648Online:01 December 1993Publication History 51citation1,798DownloadsMetricsTotal Citations51Total Downloads1,798Last 12 Months30Last 6 weeks4 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Abstract

Traditional models of input work on the assumption that inputs delivered to a system are fairly certain to have occurred as they are reported. However, a number of new input modalities, such as pen-based inputs, hand and body gesture inputs, and voice input, do not share this property. Inputs under these techniques are normally acquired by a process of recognition. As a result, each of these techniques makes mistakes and provides inputs which are approximate or uncertain. This paper considers some preliminry techniques for dialog management in the presence of this uncertainty. These techniques—including a new input model and a set of extended state machine abstractions—will explicitly model uncertainty and handle it as a normal and expected part of the input process.

This paper presents a novel methodology for viewing large graphs.The basic concept is to allow the user to interactively navigate through large graphs learning about them in appropriately small and concise pieces.An architecture is present to support graph exploration.It contains methods for building custom layout algorithms hierarchically, interactively decomposing large graphs, and creating interactive parameterized layout algorithms.As a proof of concept, examples are drawn from a working prototype that incorporates this methodology.

Article Free Access Share on A nose gesture interface device: extending virtual realities Authors: Tyson R. Henry Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZ Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZView Profile , Scott E. Hudson Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZ Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZView Profile , Andrey K. Yeatts Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZ Department of Computer Science and Biomedical Interfaces, University of Arizona, Tucson, AZView Profile , Brad A. Myers School of Computer Science, Carnegie Mellon University, Pittsburgh, PA School of Computer Science, Carnegie Mellon University, Pittsburgh, PAView Profile , Steven Feiner Department of Computer Science, Columbia University, New York, NY Department of Computer Science, Columbia University, New York, NYView Profile Authors Info & Claims UIST '91: Proceedings of the 4th annual ACM symposium on User interface software and technologyNovember 1991 Pages 65–68https://doi.org/10.1145/120782.120789Published:11 November 1991Publication History 5citation995DownloadsMetricsTotal Citations5Total Downloads995Last 12 Months76Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Workin automating the production of user interf%ce software has recently concentrated on two distinct approaches:systems that provide a direct manipulation editor for specifying user interfaces and systems that attempt to automatically generate much or all of the interface.This paper considers how a middle ground between these approaches can be constructed.It presents a technique whereby the rule-base inference methods used in many automatic generation systems can be smoothly integrated into a direct manipulation interface builder.This integration is achieved by explicitly representing the results of inference rules in the direct manipulation framework and by using semantic snapping techniques to give the user direct feedback and interactive control over the application of rules.

This paper introduces a new algorithm for incremental attribute evaluation.The algorithm is lazy: Itevaluates only theattributes that are both affected byachange andthat are directly or indirectly observable by the user.In this way, the wasted work of computing values that are never actually used is avoided.Although the algorithm is not optimal, it performs better than the standard "optimal" algorithm in cases where expensive but optional computations need to be supported.Furthermore, the algorithm does not have some of the limitations of other algorithms.It works for general attributed graphs as well as for standard attributed trees.In addition, it does not presume any special editing model, and it supports multiple change points without loss of efficiency.

This paper describes the implementation of semantic snapping — an interaction technique that provides semantic feedback at the lexical level while dragging a graphical object on the screen. Like conventional snapping, or gravity fields, semantic snapping includes a geometric component where objects in close proximity are drawn together or “snap” into position. However, semantic snapping goes further by allowing non-geometric (semantic) properties of objects to place additional constraints on snapping. Semantic snapping also provides more complex lexical feedback which reflects potential semantic consequences of a snap. This paper motivates the use of semantic snapping and describes how this technique has been implemented in a window-based toolkit. This implementation works in an adaptive manner to provide the best interactive response in situations where semantic tests are very time consuming and strain the limits of acceptable performance.

Article Integrating gesture and snapping into a user interface toolkit Share on Authors: Tyson R. Henry Department of Computer Science, University of Arizona, Tucson AZ Department of Computer Science, University of Arizona, Tucson AZView Profile , Scott E. Hudson Department of Computer Science, University of Arizona, Tucson AZ Department of Computer Science, University of Arizona, Tucson AZView Profile , Gary L. Newell Department of Computer Science, University of Arizona, Tucson AZ Department of Computer Science, University of Arizona, Tucson AZView Profile Authors Info & Claims UIST '90: Proceedings of the 3rd annual ACM SIGGRAPH symposium on User interface software and technologyAugust 1990 Pages 112–122https://doi.org/10.1145/97924.97938Online:01 August 1990Publication History 42citation757DownloadsMetricsTotal Citations42Total Downloads757Last 12 Months56Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Abstract The UNIX operating system has attained widespread popularity because of its power, flexibility and elegance. However, the common user interfaces (shells) have been criticized as being cryptic and difficult to use, especially for novices. In this paper, we present a user interface server that can be used to provide a graphical shell for UNIX. This shell helps to overcome a number of difficulties faced by both novices and experts. Although it is largely intended to assist novices, it is also designed not to become a hindrance for expert users and does not sacrifice the basic power and flexibility of UNIX. The server is designed to augment rather than replace text‐based interfaces and can be used from a variety of different interactive programs in addition to shells.

Background: Direct manipulation interfaces, such as the Macintosh desktop, often represent objects with icons [l, 41. For example, text files are represented by icons. Selection of the icon invokes an editor to view the file it represents, thus the icon not only represents the text file but also represents a text editor used to view the text file. Each type of object is viewed by using a specific program. Diagram objects, for example, require a graphical editor to view. In a typical iconic interface, icons represent a single view of an object. While the single-view single-icon model has proven very intuitive and easy to use, some objects have more than one logical view [2]. Program code can be viewed as text by using a text editor, or it can be viewed as program code by a compiler. The machine code that corresponds to the program code can be thought of as another view, thus the execution of a program can be thought of as another view of program code. This paper describes a mechanism for grouping a set of icons that depict several views of a single object into one multidimensional icon. Multidimensional icons group sets of icons, each describing a unique view of an object, into a single entity. The individual icons are projected onto the sides of a simulated cube. There are two distinct advantages to using a cube instead of displaying all the icons in a menu. The first advantage is that since cubes are very familiar objects, it is a natural mental model to think of the faces of a cube as views of the entire cube—views of the actual object. The second advantage is that by allowing rotation of the cube, several icons representing views of an object are accessible using only a fraction of the screen space required to display all the icons.

One of the problems with conventional UIMSs is that very often there is no graphical way to specify interfaces. This paper describesOPUS, the user interface editor of thePenguimsUIMS. This system allows the presentation component of graphical user interfaces to be specified interactively in a graphical notation without explicit programming. The Penguims UIMS supports an underlying model of computation based loosely on spreadsheets. In particular, it supports incremental computations based on a system of equations (one-way constraints) over a set of named values (spreadsheet cells). These equations are used to provide immediate feedback at all levels of the interface. They are used to incrementally determine the position and dynamic appearance of the individual interactor objects that make up the interface. They are also used to connect the presentation directly to underlying application data thereby supporting semantic feedback. The OPUS user interface editor employs a special graphical notation for specifying the presentation component of a user interface. This notation allows the power of the underlying computational model to be expressed simply and quickly. The resulting presentations are very flexible in nature. They can automatically respond to changes in the size and position of display objects and can directly support derivation of their appearance from application data objects.

This paper describes the implementation concepts behind the user interface editor of the Apogee UIMS. This editor allows many aspects of a user interface to be specified graphically without a conventional textual specification. The system supports the specification of flexible user interfaces — ones that can adapt automatically to changes in the size of objects they present and that can adapt to specific user needs in a dynamic and responsive fashion. To serve as an implementation base for this editor, the Apogee UIMS supports an active data model based on one-way constraints. This model is implemented by a small object-oriented programming language embedded within the system.

Cactis is an object-oriented, multiuser DBMS developed at the University of Colorado. The system supports functionally-defined data and uses techniques based on attributed graphs to optimize the maintenance of functionally-defined data. The implementation is self-adaptive in that the physical organization and the update algorithms dynamically change in order to reduce disk access. The system is also concurrent. At any given time there are some number of computations that must be performed to bring the database up to date; these computations are scheduled independently and are performed when the expected cost to do so is minimal. The DBMS runs in the Unix/C Sun workstation environment. Cactis is designed to support applications that require rich data modeling capabilities and the ability to specify functionally-defined data, but that also demand good performance. Specifically, Cactis is intended for use in the support of such applications as VLSI and PCB design, and software environments.

An active data model is one that not only stores data, but also acts upon changes to the data in ways that reflect application domain semantics. This paper introduces an active object-oriented model based on incremental attribute evaluation (one-way constraints) and discusses how it can be used to support a number of tasks in a User Interface Management System. It is shown how this model can provide unified support for lexical, syntactic, and semantic feedback, how the model supports the specification and implementation of dynamically changing graphical layouts, and how the model can be used as the basis for support of user extensible interfaces.

The Cactis project is an on-going effort oriented toward extending database support from traditional business-oriented applications to software environments. The main goals of the project are to construct an appropriate model, and develop new techniques to support the unusual data management needs of software environments, including program compilations, software configurations, load modules, project schedules, software versions, nested and long transactions, and program transformations. The ability to manage derived information is common to many of these data needs, and the Cactis database management system has the ability to represent and maintain derived data in a time- and space-efficient fashion. A central contribution of Cactis is its integration of the type constructors of semantic models and the localized behavior capabilities of object-oriented database management systems.>

Almost all applications using interactive graphics contain important structures and concepts which are deeper than the geometres used to display them to the user. One of the major tasks of the system implementer is to cause the user interface to reflect this deeper structure accurately so that it may be directly manipulated by the user. The authors describe a tool, the Higgens user interface management system (UIMS), which can automate much of this task for a wide class of systems using interactive graphics. It is able to generate graphical user interfaces automatically from a high-level interface specification. These specifications are primarily nonprocedural in nature. They describe how graphical images can be automatically derived and updated based on applications entities, and how graphical inputs can be translated back into terms which are appropriate to the application.>

Cactis is an object-oriented, multi-user DBMS developed at the University of Colorado. The implementation is self-adaptive and concurrent, and runs in the Unix/C Sun workstation environment. A central, unique focus of Cactis is the support of functionally-defined data in a manner which provides good performance. Cactis is intended for use in applications which are conducive to an object-oriented approach and involve derived data. Such applications include software environments.

A system for generating direct manipulation office systems is described. In these systems, the user directly manipulates graphical representations of office entities instead of dealing with these entities abstractly through a command language or menu system. These systems employ a new semantic data model to describe office entities. New techniques based on attribute grammars and incremental attribute evaluation are used to implement this data model in an efficient manner. In addition, the system provides a means of generating sophisticated graphics-based user interfaces that are integrated with the underlying semantic model. Finally, the generated systems contain a general user reversal and recovery (or undo) mechanism that allows them to be much more tolerant of human errors.