Examples are powerful tools for creativity and can provide inspiration and structure. However, novices often don’t know when to seek examples or how to apply them. Showing real-time examples targeted to the current task (adaptive conceptual guidance) may help novices consider inspiration more often and better implement the ideas that examples illustrate. We explore this in a Wizard-of-Oz system, Shöwn, that presents examples based on the user’s current activity while drawing a comic strip. Shöwn’s design is informed by interviews with novice and expert comic artists (n=18). A between-subjects experiment (n=24) found that adaptive conceptual guidance improved the clarity and uniqueness of drawings and stories compared to non-adaptive examples. Users also found the examples more useful and inspirational than those without adaptive guidance. Our results present an initial exploration of the challenges and benefits of contextually showing examples in interactive creativity tools.

People have scientific questions and folk theories; yet most lack the expertise to investigate them. How might people transform their questions into experiments that inform both science and their lives? This paper demonstrates how online volunteers can collaboratively design and run experiments using a novel social computing system. The Galileo system provides procedural support using three techniques: 1) experimental design workflow that provides just-in-time training; 2) review workflow with scaffolded questions; and 3) automated routines for data collection. We present two empirical investigations: a study and a field deployment with online volunteers across 16 and 8 countries respectively. People generated structurally-sound experiments on personally meaningful topics; three communities ran a week-long experiment each. We identify two key challenges for citizen-led experimentation—supporting different expertise levels and providing recruitment guidance—and provide specific suggestions from the social computing literature. Our results highlight the promise and challenges of citizen-led knowledge work like experimentation.

This paper investigates in-class interactions in synchronous online classrooms when the choice of modality is discretionary, such that students choose when and if they turn on their cameras and microphones. Instructor interviews (N = 7) revealed that most students preferred not to share videos and verbally participate. This hindered instructors’ ability to read their classrooms and make deeper connections with students. Survey results (N = 102) suggested that students felt a lacking sense of community in online vs. in-person lectures. Some students felt uncomfortable broadcasting their appearances to everyone in the class, and some were unaware of the benefits for instructors. Most students favored using the text chat to participate. Considering the needs of both instructors and students, we propose recommendations to mitigate the loss of classroom interactions by collecting and presenting less invasive social cues in an aggregated format, and incorporating opportunities for informal exchanges and individual control to spark peer bonding.

Discussion fora of instructional videos contain previously-discussed questions and answers. These video comments can resolve many points of confusion for learners. However, finding relevant content in a separated discussion forum is challenging and disruptive to learning flow. This paper introduces Adjacent Display of Relevant Discussion (ADRD): it displays threaded comments in a panel adjacent to the video and dynamically updates the content of the panel based on the time of the video. In a between-subjects lab study (n=20), ADRD helped participants resolve confusion points, skim and read comments, and encouraged video interaction.

People often seek help online while using complex software. Currently, information search takes users' attention away from the task at hand by creating a separate search task. This paper investigates how multimodal interaction can make in-task help-seeking easier and faster. We introduce ReMap, a multimodal search interface that helps users find video assistance while using desktop and web applications. Users can speak search queries, add application-specific terms deictically (e.g., "how to erase this"), and navigate search results via speech, all without taking their hands (or mouse) off their current task. Thirteen participants who used ReMap in the lab found that it helped them stay focused on their task while simultaneously searching for and using learning videos. Users' experiences with ReMap also raised a number of important challenges with implementing system-wide context-aware multimodal assistance.

Many artists livestream their creative process, allowing viewers to learn and be inspired from the decisions -- and mistakes -- they make along the way. This paper presents the first broad look at the range of creative activities people stream. Through content analysis of livestream archives, interviews with 8 streamers, and online surveys with 165 viewers, we study current practices and challenges in creative livestream communities and compare them with prior observations of livestreaming in other domains. We observed four common types of creative livestreams: teaching, making, socializing, and performing. We identify three open questions for the research community around how to better support the goals of creative streamers and viewers: how to support richer audience interactions at scale, how to support all parts of the creative process, and how to support watching livestream archives.

Misunderstandings of science affect many lives. Novices commonly misunderstand explanations by overly relying on surface details instead of evaluating underlying logic. Prior work has found adding a patina of neuroscience leads readers towards positively assessing explanations. How might we help people better understand science explanations? A between-subjects experiment tested whether asking readers to reconstruct experiments leads them to focus more on underlying logic. Participants relied less on irrelevant surface details when reconstructing experiments. However, this did not impact their subsequent assessment of explanations. Our results suggest that reconstruction is a useful strategy for understanding explanations but is not readily transferred towards evaluating explanations.

Effective creative work requires both "hot" (exploratory) and "cool" (exploitative) thinking. Unfortunately, many people (especially novices) under-explore, jumping to the "cool'' part too quickly, because they assume their current thinking "has to be" the path. This paper presents empirical results of how metaphorical problem framing scaffolds can influence creative performance. The task used De Bono's "Thinking Hats." In a between-subjects experiment comparing exploratory to exploitative problem frames, the exploratory problem frame led to more original designs and more diverse ideas during brainstorming. This work provides an empirical baseline of how -- even for short tasks -- assigning people responsibility for broad thinking leads to better creative work.

Complex activities often require people to work across multiple software applications. However, people frequently lack valuable knowledge about at least one application, especially as software changes and new software emerges. Existing help systems either lack contextual knowledge or are tightly-knit into a single application. We introduce an application-independent approach for contextually presenting video learning resources and demonstrate it through the RePlay system. RePlay uses accessibility APIs to gather context about the user's activity. It leverages an existing search engine to present relevant videos and highlights key segments within them using video captions. We report on a week-long field study (n=7) and a lab study (n=24) showing that contextual assistance helps people spend less time away from their task than web video search and replaces current video navigation strategies. Our findings highlight challenges with representing and using context across applications.

Peer review asks novices to take on an evaluator's role, yet novices often lack the perspective to accurately assess the quality of others' work. To help learners give feedback on their peers' work through an expert lens, we present the Juxtapeer peer review system for structured comparisons. We build on theories of learning through contrasting cases, and contribute the first systematic evaluation of comparative peer review. In a controlled experiment, 476 consenting learners across four courses submitted 1,297 submissions, 4,102 reviews, and 846 self assessments. Learners assigned to compare submissions wrote reviews and self-reflections that were longer and received higher ratings from experts than those who evaluated submissions one at a time. A second study found that a ranking of submissions derived from learners' comparisons correlates well with staff ranking. These results demonstrate that comparing algorithmically-curated pairs of submissions helps learners write better feedback.

Good feedback is critical to creativity and learning, yet rare. Many people do not know how to actually provide effective feedback. There is increasing demand for quality feedback -- and thus feedback givers -- in learning and professional settings. This paper contributes empirical evidence that two interactive techniques -- reusable suggestions and adaptive guidance -- can improve feedback on creative work. We present these techniques embodied in the CritiqueKit system to help reviewers give specific, actionable, and justified feedback. Two real-world deployment studies and two controlled experiments with CritiqueKit found that adaptively-presented suggestions improve the quality of feedback from novice reviewers. Reviewers also reported that suggestions and guidance helped them describe their thoughts and reminded them to provide effective feedback.

People's lived experiences provide intuitions about health. Can they transform these personal intuitions into testable hypotheses that could inform both science and their lives? This paper introduces an online learning architecture and provides system principles for people to brainstorm causal scientific theories. We describe the Learn-Train-Ask workflow that guides participants through learning domain-specific content, process training to frame their intuitions as hypotheses, and collaborating with anonymous peers to brainstorm related questions. 344 voluntary online participants from 27 countries created 399 personally-relevant questions about the human microbiome over 4 months, 75 (19%) of which microbiome experts found potentially scientifically novel. Participants with access to process training generated hypotheses of better quality. Access to learning materials improved the questions' microbiome-specific knowledge. These results highlight the promise of performing personally-meaningful scientific work using massive online learning systems.

How does group membership framing affect the feedback students provide learners? This paper presents two between-subjects experiments investigating the effect of Ingroup/Outgroup membership on effort spent in peer evaluations, and whether the group membership criterion type affects quality and stringency of evaluation. Two peer-review assignments were implemented in two separate classes. In the first study, students were nominally grouped by location they sat in class and non-nominally grouped by current class score; each was asked to review an Ingroup and Outgroup peer assignment. A second study randomly assigned students to one of four group types (random, score, motivation, and location); student reviewed two Ingroup assignments. In both studies, score-grouped students graded their peers more stringently than students grouped by location. These studies illustrate for system designers the impacts of group framing - and the disclosure of that-in peer review tasks.

Email has scaled our ability to communicate with large groups, but has not equivalently scaled our ability to listen and respond. For example, emailing many people for feedback requires either impersonal surveys or manual effort to hold many similar conversations. To scale personalized conversations, we introduce techniques that exploit similarities across conversations to recycle relevant parts of previous conversations. These techniques reduce the authoring burden, save senders' time, and maintain recipient engagement through personalized responses. We introduce MyriadHub, a mail client where users start conversations and then crowd workers extract underlying conversational patterns and rules to accelerate responses to future similar emails. In a within-subjects experiment comparing MyriadHub to existing mass email techniques, senders spent significantly less time planning events with MyriadHub. In a second experiment comparing MyriadHub to a standard email survey, MyriadHub doubled the recipients' response rate and tripled the number of words in their responses.

Learners worldwide collectively spend millions of hours per week testing their skills on assignments with known answers. Might some of this time fruitfully be spent posing and exploring novel questions? This paper investigates an approach for learners to contribute scientific ideas. The Gut Instinct system embodies this approach, hosting online learning materials and invites learners to collaboratively brainstorm potential influences on people's microbiome. A between-subjects experiment compared the performance of participants who engaged in just learning, just contributing, or a combination. Participants in the learning condition scored highest on a summative test. Participants in both the contribution and combined conditions generated novel, useful questions; there was not a significant difference between the two. Though participants in the combined condition both learned and contributed, this setting did not exhibit an additive benefit, such as better learning in the combined condition. These results highlight the promise and difficulty of double-bottom-line learning experiences.

Many studies demonstrate that peer reviewing provides pedagogical benefits such as inspiration and developing expert vision, and changes classroom culture by encouraging reciprocity. However, much large-scale research in peer assessment has focused on MOOCs, where students have short tenures, and is unable to describe how reciprocity-oriented classroom cultures evolve over time. This short paper presents the first long-term analysis of peer reviewing with 304 students, conducted in three large physical classes in a year-long undergraduate series. Surprisingly, this analysis reveals that when students receive better reviews on their work, they write worse reviews in the future. This suggests that while students believe in the reciprocal nature of peer review, they act anti-reciprocally. Therefore, battling the emergent norm of anti-reciprocity is crucial both for system designers and practitioners who use peer assessment.

Complex software offers power for experts, yet overwhelms new users. Novices often do not know how to execute tasks, what they want to achieve, or even what is possible. To address this, we introduce the DiscoverySpace interface for executable action suggestions. DiscoverySpace is a prototype extension panel for Adobe Photoshop that suggests task-level action macros to apply to photographs based on visual features. DiscoverySpace harvests these one-click actions from the online Photoshop user community. A between-subjects study indicated that action suggestions may help novices maintain confidence, accomplish tasks, and discover features. This work demonstrates how interfaces can leverage user-generated content to help novices navigate complex software.

Peer assessment is rapidly growing in online learning, as it presents a method to address scalability challenges. However, research suggests that the benefits of peer review are obtained inconsistently. This paper explores why, introducing three ways that framing task goals significantly changes reviews. Three experiments manipulated features in the review environment. First, adding a numeric scale to open text reviews was found to elicit more explanatory, but lower quality reviews. Second, structuring a review task into short, chunked stages elicited more diverse feedback. Finally, showing reviewers a draft along with finished work elicited reviews that focused more on the work's goals than aesthetic details. These findings demonstrate the importance of carefully structuring online learning environments to ensure high quality peer reviews.

This work examines whether templates designed from principles of multimedia learning design, and learning sciences research, can support peer instruction in creating more effective educational content on the web. Initial results show that the structure and guidelines within these templates can help novices produce meaningful learning content while improving the overall learning experience. This experiment provides insights into how to design and implement structured outlines online for web users to share learning content, and potentially shift researchers' focus to more learner-centered online education.

Massive online classes are global and diverse. How can we harness this diversity to improve engagement and learning? Currently, though enrollments are high, students' interactions with each other are minimal: most are alone together. This isolation is particularly disappointing given that a global community is a major draw of online classes. This paper illustrates the potential of leveraging geographic diversity in massive online classes. We connect students from around the world through small-group video discussions. Our peer discussion system, Talkabout, has connected over 5,000 students in fourteen online classes. Three studies with 2,670 students from two classes found that globally diverse discussions boost student performance and engagement: the more geographically diverse the discussion group, the better the students performed on later quizzes. Through this work, we challenge the view that online classes are useful only when in-person classes are unavailable. Instead, we demonstrate how diverse online classrooms can create benefits that are largely unavailable in a traditional classroom.

Research suggests that online peer review can provide critical help to learners who would otherwise not be given individualized feedback on their work. However, little is known about how different characteristics of review systems impact reviewers. This extended abstract presents preliminary results from an online experiment examining how explicit numeric ratings change peer reviews. A between-subject experiment found that peer reviewers who were asked to generate a numeric rating as well as general feedback gave significantly more explanations and made more positive comments compared with reviewers who were asked to give general feedback only. These exploratory findings suggest the need to further examine how online peer review systems' affordances can impact the reviews given in these systems.

When students work with peers, they learn more actively, build richer knowledge structures, and connect material to their lives. However, not every peer learning experience online sees successful adoption. This paper articulates and addresses three adoption challenges for global-scale peer learning. First, peer interactions struggle to bootstrap critical mass. However, class incentives can signal importance and spur initial usage. Second, online classes have limited peer visibility and awareness, so students often feel alone even when surrounded by peers. We find that highlighting interdependence and strengthening norms can mitigate this issue. Third, teachers can readily access "big" aggregate data but not "thick" contextual data that helps build intuitions, so software should guide teachers' scaffolding of peer interactions. We illustrate these challenges through studying 8,500 students' usage of two peer learning platforms, Talkabout and PeerStudio. This paper measures efficacy through sign-up and participation rates and the structure and duration of student interactions.

Rapid feedback is a core component of mastery learning, but feedback on open-ended work requires days or weeks in most classes today. This paper introduces PeerStudio, an assessment platform that leverages the large number of students' peers in online classes to enable rapid feedback on in-progress work. Students submit their draft, give rubric-based feedback on two peers' drafts, and then receive peer feedback. Students can integrate the feedback and repeat this process as often as they desire. In MOOC deployments, the median student received feedback in just twenty minutes. Rapid feedback on in-progress work improves course outcomes: in a controlled experiment, students' final grades improved when feedback was delivered quickly, but not if delayed by 24 hours. More than 3,600 students have used PeerStudio in eight classes, both massive and in-person. This research demonstrates how large classes can leverage their scale to encourage mastery through rapid feedback and revision.

Student discussions over video in massive classes allow students to explore course content, share personal experiences and get feedback on their ideas. However, such discussions frequently turn into casual conversations without focusing on the curriculum and the learning objectives. This short paper explores whether students can achieve multiple learning objectives by solving challenges collaboratively during discussions. We introduce `think-pair-share' technique for video discussions. Our pilot results, drawn from a Coursera class, suggest that participants prefer to exchange information with their peers using personal stories and connecting stories with curriculum increases participant engagement.

research-article Share on Introduction to Online Learning at Scale Editors: Benjamin B. Bederson View Profile , Daniel M. Russell View Profile , Scott Klemmer View Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 22Issue 2April 2015 Article No.: 5pp 1–3https://doi.org/10.1145/2737794Published:15 April 2015Publication History 6citation477DownloadsMetricsTotal Citations6Total Downloads477Last 12 Months28Last 6 weeks2 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my Alerts New Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Checklist usage can increase performance in complex, high-risk domains. While paper checklists are valuable, they are static, slow to access, and show both too much and too little information. We introduce Dynamic Procedure Aids to address four key problems in checklist usage: ready access to aids, rapid assimilation of content, professional acceptance, and limited attention. To understand their efficacy for crisis response, we created the dpAid software system. Its design arose through a multi-year participation in medical crisis response training featuring realistic team simulations. A study comparing Dynamic Procedure Aids, paper, and no aid, found that participants with Dynamic Procedure Aids performed significantly better than with paper or no aid. This study introduces the narrative simulation paradigm for comparatively assessing expert procedural performance through a score-and-correct approach.

In the physical classroom, peer interactions motivate students and expand their perspective. We suggest that synchronous peer interaction can benefit massive online courses as well. Talkabout organizes students into video discussion groups and allows instructors to determine group composition and discussion content. Using Talkabout, students pick a discussion time that suits their schedule. The system groups the students into small video discussions based on instructor preferences such as gender or geographic balance. To date, 2,474 students in five massive online courses have used Talkabout to discuss topics ranging from prejudice to organizational theory. Talkabout discussions are diverse: in one course, the median six-person discussion group had students from four different countries. Students enjoyed discussing in these diverse groups: the average student participated for 66 minutes, twice the course requirement. Students in more geographically distributed groups also scored higher on the final, suggesting that distributed discussions have educational value.

Peer assessment helps students reflect and exposes them to different ideas. It scales assessment and allows large online classes to use open-ended assignments. However, it requires students to spend significant time grading. How can we lower this grading burden while maintaining quality? This paper integrates peer and machine grading to preserve the robustness of peer assessment and lower grading burden. In the identify-verify pattern, a grading algorithm first predicts a student grade and estimates confidence, which is used to estimate the number of peer raters required. Peers then identify key features of the answer using a rubric. Finally, other peers verify whether these feature labels were accurately applied. This pattern adjusts the number of peers that evaluate an answer based on algorithmic confidence and peer agreement. We evaluated this pattern with 1370 students in a large, online design class. With only 54% of the student grading time, the identify-verify pattern yields 80-90% of the accuracy obtained by taking the median of three peer scores, and provides more detailed feedback. A second experiment found that verification dramatically improves accuracy with more raters, with a 20% gain over the peer-median with four raters. However, verification also leads to lower initial trust in the grading system. The identify-verify pattern provides an example of how peer work and machine learning can combine to improve the learning experience.

Massive online courses introduced Community TAs (CTAs) to help scale teaching staff support. CTAs are former top students who return as volunteer course staff. We studied CTAs in 3 classes on Coursera, including interviews and surveys from a Human-Computer Interaction (HCI) class. A key benefit of CTAs is their brokering role that mediates staff and student goals. CTAs provide greater discussion forum coverage (both in quantity and time of day) compared to instructor and Head TA (HTA) capabilities and contribute to peer assessment. As CTAs are new teachers, physically distributed, and culturally diverse, clear division of responsibilities is especially important.

The design of artifacts and how designers make them have garnered renewed societal interest as interactive technologies create new opportunities and challenges. The world we experience has never be...

Advances in data mining and knowledge discovery have transformed the way Web sites are designed. However, while visual presentation is an intrinsic part of the Web, traditional data mining techniques ignore render-time page structures and their attributes. This paper introduces design mining for the Web: using knowledge discovery techniques to understand design demographics, automate design curation, and support data-driven design tools. This idea is manifest in Webzeitgeist, a platform for large-scale design mining comprising a repository of over 100,000 Web pages and 100 million design elements. This paper describes the principles driving design mining, the implementation of the Webzeitgeist architecture, and the new class of data-driven design applications it enables.

This paper introduces privacy and accountability techniques for crowd-powered systems. We focus on email task management: tasks are an implicit part of every inbox, but the overwhelming volume of incoming email can bury important requests. We present EmailValet, an email client that recruits remote assistants from an expert crowdsourcing marketplace. By annotating each email with its implicit tasks, EmailValet's assistants create a task list that is automatically populated from emails in the user's inbox. The system is an example of a valet approach to crowdsourcing, which aims for parsimony and transparency in access con-trol for the crowd. To maintain privacy, users specify rules that define a sliding-window subset of their inbox that they are willing to share with assistants. To support accountability, EmailValet displays the actions that the assistant has taken on each email. In a weeklong field study, participants completed twice as many of their email-based tasks when they had access to crowdsourced assistants, and they became increasingly comfortable sharing their inbox with assistants over time.

Designers in many fields rely on examples for inspiration, and examples play an important role in art and design curricula. Online media offer a corpus of examples at a scale and diversity never before seen. Design on the Web is not only large, it's also “at large.” This scale and diversity enable powerful new opportunities for learning, mining, understanding, and assessing design. My group's research tools harvest and synthesize examples to empower more people to design interactive systems, learners to acquire new skills, experts to be more creative, and programmers to engage in more design thinking. This research shapes my project-based design teaching, which emphasizes creating diverse alternatives, self-assessment, and using examples to provide design insights and teach abstract principles. To illustrate these opportunities — and the technical and social challenges — of design that is large and at-large, I'll focus on two recent projects in my group. The first explores peer assessment of design at a global scale. In 2012, we collaborated with Coursera to launch the first massive-scale class with self and peer assessment. This enabled online students to engage in open-ended design projects. It has also worked surprisingly well, and variants have since been used by more than 80 other massive online classes. The second project, Webzeitgeist, introduces a scalable platform for Web design mining. Applications built with this platform can dynamically curate design galleries, search for design alternatives, retarget content between page designs, and predict the semantic role of page elements from design data. In sharing this work, I'll try to impart some useful strategies and spark discussion on design research at large.

People complete tasks more quickly when they have concrete plans. However, they often fail to create such action plans. (How) can systems provide these concrete steps automatically? This article demonstrates that these benefits can also be realized when these plans are created by others or reused from similar tasks. Four experiments test these approaches, finding that people indeed complete more tasks when they receive externally-created action plans. To automatically provide plans, we introduce the Genies workflow that combines benefits of crowd wisdom, collaborative refinement, and automation. We demonstrate and evaluate this approach through the TaskGenies system, and introduce an NLP similarity algorithm for reusing plans. We demonstrate that it is possible for people to create action plans for others, and we show that it can be cost effective.

Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9% of students’ grades within 5% of the staff grade, and 65.5% within 10%. On average, students assessed their work 7% higher than staff did. Students also rated peers’ work from their own country 3.6% higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4% to 9.9%.

Micro-task platforms provide massively parallel, on-demand labor. However, it can be difficult to reliably achieve high-quality work because online workers may behave irresponsibly, misunderstand the task, or lack necessary skills. This paper investigates whether timely, task-specific feedback helps crowd workers learn, persevere, and produce better results. We investigate this question through Shepherd, a feedback system for crowdsourced work. In a between-subjects study with three conditions, crowd workers wrote consumer reviews for six products they own. Participants in the None condition received no immediate feedback, consistent with most current crowdsourcing practices. Participants in the Self-assessment condition judged their own work. Participants in the External assessment condition received expert feedback. Self-assessment alone yielded better overall work than the None condition and helped workers improve over time. External assessment also yielded these benefits. Participants who received external assessment also revised their work more. We conclude by discussing interaction and infrastructure approaches for integrating real-time assessment into online work.

Online communities enable people to easily connect and share knowledge across geographies. Mobile phones can enable billions of new users in emerging countries to participate in these online communities. In India, where social hierarchy is important, users may over-value institutionally-recognized authorities relative to peer-sourced content. We tested this hypothesis through a controlled experiment of source authority effects on a voice-based agricultural information service for farmers in Gujarat, India. 305 farmers were sent seven agricultural tips via automated phone calls over a two-week period. The same seven tips were each voice-recorded by two university scientists and two peer farmers. Participants received a preview of the tip from a randomly assigned source via the automated call, and played the remainder of the tip by calling a dedicated phone number. Participants called the follow-up number significantly more often when the tip preview was recorded by a peer than a scientist. On the other hand, in interviews conducted both before and after the experiment, a majority of farmers maintained that they preferred receiving information from scientists. This stated preference may have been expressing the more socially acceptable response. We interpret our experimental results as a demonstration of the demand for peer-based agricultural information dissemination. We conclude with design implications for peer-to-peer information services for rural communities in India.

Prototypes ground group communication and facilitate decision making. However, overly investing in a single design idea can lead to fixation and impede the collaborative process. Does sharing multiple designs improve collaboration? In a study, participants created advertisements individually and then met with a partner. In the Share Multiple condition, participants designed and shared three ads. In the Share Best condition, participants designed three ads and selected one to share. In the Share One condition, participants designed and shared one ad. Sharing multiple designs improved outcome, exploration, sharing, and group rapport. These participants integrated more of their partner's ideas into their own subsequent designs, explored a more divergent set of ideas, and provided more productive critiques of their partner's designs. Furthermore, their ads were rated more highly and garnered a higher click-through rate when hosted online.

The Web provides a corpus of design examples unparalleled in human history. However, leveraging existing designs to produce new pages is often difficult. This paper introduces the Bricolage algorithm for transferring design and content between Web pages. Bricolage employs a novel, structured-prediction technique that learns to create coherent mappings between pages by training on human-generated exemplars. The produced mappings are then used to automatically transfer the content from one page into the style and layout of another. We show that Bricolage can learn to accurately reproduce human page mappings, and that it provides a general, efficient, and automatic technique for retargeting content between a variety of real Web pages.

In design, people often seek examples for inspiration. However, current example-finding practices suffer many drawbacks: templates present designs without a usage context; search engines can only examine the text on a page. This paper introduces exploratory techniques for finding relevant and inspiring design examples. These novel techniques include searching by stylistic similarity to a known example design and searching by stylistic keyword. These interactions are manifest in d.tour, a style-based design exploration tool. d.tour presents a curated database of Web pages as an explorable design gallery. It extracts and analyzes design features of these pages, allowing it to process style-based queries and recommend designs to the user. d.tour's gallery interface decreases the gulfs of execution and evaluation for design example-finding.

The ready availability of online source-code examples has fundamentally changed programming practices. However, current search tools are not designed to assist with programming tasks and are wholly separate from editing tools. This paper proposes that embedding a task-specific search engine in the development environment can significantly reduce the cost of finding information and thus enable programmers to write better code more easily. This paper describes the design, implementation, and evaluation of Blueprint, a Web search interface integrated into the Adobe Flex Builder development environment that helps users locate example code. Blueprint automatically augments queries with code context, presents a code-centric view of search results, embeds the search experience into the editor, and retains a link between copied code and its source. A comparative laboratory study found that Blueprint enables participants to write significantly better code and find example code significantly faster than with a standard Web browser. Analysis of three months of usage logs with 2,024 users suggests that task-specific search interfaces can significantly change how and when people search the Web.

Interaction designers typically revise user interface prototypes by adding unstructured notes to storyboards and screen printouts. How might computational tools increase the efficacy of UI revision? This paper introduces d.note, a revision tool for user interfaces expressed as control flow diagrams. d.note introduces a command set for modifying and annotating both appearance and behavior of user interfaces; it also defines execution semantics so proposed changes can be tested immediately. The paper reports two studies that compare production and interpretation of revisions in d.note to freeform sketching on static images (the status quo). The revision production study showed that testing of ideas during the revision process led to more concrete revisions, but that the tool also affected the type and number of suggested changes. The revision interpretation study showed that d.note revisions required fewer clarifications, and that additional techniques for expressing revision intent could be beneficial.

Interpreting compiler errors and exception messages is challenging for novice programmers. Presenting examples of how other programmers have corrected similar errors may help novices understand and correct such errors. This paper introduces HelpMeOut, a social recommender system that aids the debugging of error messages by suggesting solutions that peers have applied in the past. HelpMeOut comprises IDE instrumentation to collect examples of code changes that fix errors; a central database that stores fix reports from many users; and a suggestion interface that, given an error, queries the database for a list of relevant fixes and presents these to the programmer. We report on implementations of this architecture for two programming languages. An evaluation with novice programmers found that the technique can suggest useful fixes for 47% of errors after 39 person-hours of programming in an instrumented environment.

Designers often use examples for inspiration; examples offer contextualized instances of how form and content integrate. Can interactive example galleries bring this practice to everyday users doing design work, and does working with examples help the designs they create? This paper explores whether people can realize significant value from explicit mechanisms for designing by example modification. We present the results of three studies, finding that independent raters prefer designs created with the aid of examples, that examples may benefit novices more than experienced designers, that users prefer adaptively selected examples to random ones, and that users make use of multiple examples when creating new designs. To enable these studies and demonstrate how software tools can facilitate designing with examples, we introduce interface techniques for browsing and borrowing from a corpus of examples, manifest in the Adaptive Ideas Web design tool. Adaptive Ideas leverages a faceted metadata interface for viewing and navigating example galleries.

Iteration can help people improve ideas. It can also give rise to fixation, continuously refining one option without considering others. Does creating and receiving feedback on multiple prototypes in parallel, as opposed to serially, affect learning, self-efficacy, and design exploration? An experiment manipulated whether independent novice designers created graphic Web advertisements in parallel or in series. Serial participants received descriptive critique directly after each prototype. Parallel participants created multiple prototypes before receiving feedback. As measured by click-through data and expert ratings, ads created in the Parallel condition significantly outperformed those from the Serial condition. Moreover, independent raters found Parallel prototypes to be more diverse. Parallel participants also reported a larger increase in task-specific self-confidence. This article outlines a theoretical foundation for why parallel prototyping produces better design results and discusses the implications for design education.

Iterative prototyping helps designers refine their ideas and discover previously unknown issues and opportunities. However, the time constraints of production schedules can discourage iteration in favor of realization. Is this tradeoff prudent? This paper investigates if - under tight time constraints - iterating multiple times provides more benefit than a single iteration. A between-subjects study manipulates participants' ability to iterate on a design task. Participants in the iteration condition outperformed those in the non-iteration condition. Participants with prior experience with the task performed better. Notably, participants in the iteration condition without prior task experience performed as well as non-iterating participants with prior task experience.

This paper investigates the role of online resources in problem solving. We look specifically at how programmers - an exemplar form of knowledge workers - opportunistically interleave Web foraging, learning, and writing code. We describe two studies of how programmers use online resources. The first, conducted in the lab, observed participants' Web use while building an online chat room. We found that programmers leverage online resources with a range of intentions: They engage in just-in-time learning of new skills and approaches, clarify and extend their existing knowledge, and remind themselves of details deemed not worth remembering. The results also suggest that queries for different purposes have different styles and durations. Do programmers' queries "in the wild" have the same range of intentions, or is this result an artifact of the particular lab setting? We analyzed a month of queries to an online programming portal, examining the lexical structure, refinements made, and result pages visited. Here we also saw traits that suggest the Web is being used for learning and reminding. These results contribute to a theory of online resource usage in programming, and suggest opportunities for tools to facilitate online knowledge work.

How is work created, assigned, and completed on large-scale, crowd-powered systems like Wikipedia? And what design principles might enable these federated online systems to be more effective? This paper reports on a qualitative study of work and task practices on Wikipedia. Despite the availability of tag-based community-wide task assignment mechanisms, informants reported that self-directed goals, within-topic expertise, and fortuitous discovery are more frequently used than community-tagged tasks. We examine how Wikipedia editors organize their actions and the actions of other participants, and what implications this has for understanding, and building tools for, crowd-powered systems, or any web site where the main

There is great potential in enabling users to interact with digital information by integrating it with everyday physical objects. However, developing these interfaces requires programmers to acquire and abstract physical input. This is difficult, is time-consuming, and requires a high level of technical expertise in fields very different from user interface development—especially in the case of computer vision. Based on structured interviews with researchers, a literature review, and our own experience building physical interfaces, we created Papier-Mâché, a toolkit for integrating physical and digital interactions. Its library supports computer vision, electronic tags, and barcodes. Papier-Mâché introduces high-level abstractions for working with these input technologies that facilitate technology portability. We evaluated this toolkit through a laboratory study and longitudinal use in course and research projects, finding the input abstractions, technology portability, and monitoring facilities to be highly effective.

An important challenge in designing ubiquitous computing experiences is negotiating transitions between explicit and implicit interaction, such as how and when to provide users with notifications. While the paradigm of implicit interaction has important benefits, it is also susceptible to difficulties with hidden modes, unexpected action, and misunderstood intent. To address these issues, this work presents a framework for implicit interaction and applies it to the design of an interactive whiteboard application called Range. Range is a public interactive whiteboard designed to support co-located, ad-hoc meetings. It employs proximity sensing capability to proactively transition between display and authoring modes, to clear space for writing, and to cluster ink strokes. We show how the implicit interaction techniques of user reflection (how systems indicate to users what they perceive or infer), system demonstration (how systems indicate what they are doing), and override (how users can interrupt or stop a proactive system action) can prevent, mitigate, and correct errors in the whiteboard's proactive behaviors. These techniques can be generalized to improve the designs of a wide array of ubiquitous computing experiences.

Creating multiple prototypes facilitates comparative reasoning, grounds team discussion, and enables situated exploration. However, current interface design tools focus on creating single artifacts. This paper introduces the Juxtapose code editor and runtime environment for designing multiple alternatives of both application logic and interface parameters. For rapidly comparing code alternatives, Juxtapose introduces selectively parallel source editing and execution. To explore parameter variations, Juxtapose automatically creates control interfaces for "tuning" application variables at runtime. This paper describes techniques to support design exploration for desktop, mobile, and physical interfaces, and situates this work in a larger design space of tools for explorative programming. A summative study of Juxtapose with 18 participants demonstrated that parallel editing and execution are accessible to interaction designers and that designers can leverage these techniques to survey more options, faster.

This paper explores architectural support for interfaces combining pen, paper, and PC. We show how the eventbased approach common to GUIs can apply to augmented paper, and describe additions to address paper’s distinguishing characteristics. To understand the developer experience of this architecture, we deployed the toolkit to 17 student teams for six weeks. Analysis of the developers ’ code provided insight into the appropriateness of events for paper UIs. The usage patterns we distilled informed a second iteration of the toolkit, which introduces techniques for integrating interactive and batched input handling, coordinating interactions across devices, and debugging paper applications. The study also revealed that programmers created gesture handlers by composing simple ink measurements. This desire for informal interactions inspired us to include abstractions for recognition. This work has implications beyond paper — designers of graphical tools can examine API usage to inform iterative toolkit development.

Over the past decade and a half, corporations and academies have invested considerable time and money in the realization of ubiquitous computing. Yet design approaches that yield ecologically valid understandings of ubiquitous computing systems, which can help designers make design decisions based on how systems perform in the context of actual experience, remain rare. The central question underlying this article is, What barriers stand in the way of real-world, ecologically valid design for ubicomp? Using a literature survey and interviews with 28 developers, we illustrate how issues of sensing and scale cause ubicomp systems to resist iteration, prototype creation, and ecologically valid evaluation. In particular, we found that developers have difficulty creating prototypes that are both robust enough for realistic use and able to handle ambiguity and error and that they struggle to gather useful data from evaluations because critical events occur infrequently, because the level of use necessary to evaluate the system is difficult to maintain, or because the evaluation itself interferes with use of the system. We outline pitfalls for developers to avoid as well as practical solutions, and we draw on our results to outline research challenges for the future. Crucially, we do not argue for particular processes, sets of metrics, or intended outcomes, but rather we focus on prototyping tools and evaluation methods that support realistic use in realistic settings that can be selected according to the needs and goals of a particular developer or researcher.

Web designers use pens, paper, walls, and tables for explaining, developing, and communicating ideas during the early phases of design. These practices inspired The Designers' Outpost. With Outpost, users collaboratively author Web site information architectures on an electronic whiteboard using physical media (sticky notes and images), structuring and annotating that information with electronic pens. This interaction is enabled by a touch-sensitive electronic whiteboard augmented with a computer vision system. The Designers' Outpost integrates wall-scale, paper-based design practices with novel electronic tools to better support collaboration during early-phase design. Our studies with professional designers showed this integration to be especially helpful for fluidly transitioning to other design tools, access and exploration of design history, and remote collaboration.

Sensors are becoming increasingly important in interaction design. Authoring a sensor-based interaction comprises three steps: choosing and connecting the appropriate hardware, creating application logic, and specifying the relationship between sensor values and application logic. Recent research has successfully addressed the first two issues. However, linking sensor input data to application logic remains an exercise in patience and trial-and-error testing for most designers. This paper introduces techniques for authoring sensor-based interactions by demonstration. A combination of direct manipulation and pattern recognition techniques enables designers to control how demonstrated examples are generalized to interaction rules. This approach emphasizes design exploration by enabling very rapid iterative demonstrate-edit-review cycles. This paper describes the manifestation of these techniques in a design tool, Exemplar, and presents evaluations through a first-use lab study and a theoretical analysis using the Cognitive Dimensions of Notation framework.

Source-code examples of APIs enable developers to quickly gain a gestalt understanding of a library's functionality, and they support organically creating applications by incrementally modifying a functional starting point. As an increasing number of web sites provide APIs, significantlatent value lies in connecting the complementary representations between site and service - in essence, enabling sites themselves to be the example corpus. We introduce d.mix, a tool for creating web mashups that leverages this site-to-service correspondence. With d.mix, users browse annotated web sites and select elements to sample. d.mix's sampling mechanism generates the underlying service calls that yield those elements. This code can be edited, executed, and shared in d.mix's wiki-based hosting environment. This sampling approach leverages pre-existing web sites as example sets and supports fluid composition and modification of examples. An initial study with eight participants found d.mix to enable rapid experimentation, and suggested avenues for improving its annotation mechanism.

Our physical bodies play a central role in shaping human experience in the world, understandingof the world, and interactions in the world. This paper draws on theories of embodiment - from psychology, sociology, and philosophy - synthesizing five themes we believe are particularly salient for interaction design: thinking through doing, performance, visibility, risk, and thick practice. We intro-duce aspects of human embodied engagement in the world with the goal of inspiring new interaction design ap-proaches and evaluations that better integrate the physical and computational worlds.

As our business, academic, and personal lives continue to move at an ever-faster pace, finding times for busy people to meet has become an art. One of the most perplexing challenges facing groupware is effective asynchronous group scheduling (GS). This paper presents a lightweight interaction model for GS that can extend its reach beyond users of current group calendaring solutions. By expressing availability in terms of preferences, we create a flexible framework for GS that preserves plausible deniability while exerting social pressure to encourage honesty among users. We also propose an ontology that enables us to model user preferences with machine learning, predicting user responses to further lower cognitive load. The combination of visualization/direct manipulation with machine learning allows users to easily and efficiently optimize meeting times. We also suggest resulting design implications for this class of intelligent user interfaces.

Through a study of field biology practices, we observed that biology fieldwork generates a wealth of heterogeneous information, requiring substantial labor to coordinate and distill. To manage this data, biologists leverage a diverse set of tools, organizing their effort in paper notebooks. These observations motivated ButterflyNet, a mobile capture and access system that integrates paper notes with digital photographs captured during field research. Through ButterflyNet, the activity of leafing through a notebook expands to browsing all associated digital photos. ButterflyNet also facilitates the transfer of captured content to spreadsheets, enabling biologists to share their work. A first-use study with 14 biologists found this system to offer rich data capture and transformation, in a manner felicitous with current practice.

Prototyping is the pivotal activity that structures innovation, collaboration, and creativity in design. Prototypes embody design hypotheses and enable designers to test them. Framin design as a thinking-by-doing activity foregrounds iteration as a central concern. This paper presents d.tools, a toolkit that embodies an iterative-design-centered approach to prototyping information appliances. This work offers contributions in three areas. First, d.tools introduces a statechart-based visual design tool that provides a low threshold for early-stage prototyping, extensible through code for higher-fidelity prototypes. Second, our research introduces three important types of hardware extensibility - at the hardware-to-PC interface, the intra-hardware communication level, and the circuit level. Third, d.tools integrates design, test, and analysis of information appliances. We have evaluated d.tools through three studies: a laboratory study with thirteen participants; rebuilding prototypes of existing and emerging devices; and by observing seven student teams who built prototypes with d.tools.

Tangible user interfaces (TUIs) augment the physical world by integrating digital information with everyday physical objects. Currently, building these UIs requires "getting down and dirty" with input technologies such as computer vision. Consequently, only a small cadre of technology experts can currently build these UIs. Based on a literature review and structured interviews with nine TUI researchers, we created Papier-Mâché, a toolkit for building tangible interfaces using computer vision, electronic tags, and barcodes. Papier-Mache introduces a high-level event model for working with these technologies that facilitates technology portability. For example, an application can be prototyped with computer vision and deployed with RFID. We present an evaluation of our toolkit with six class projects and a user study with seven programmers, finding the input abstractions, technology portability, and monitoring window to be highly effective.

A tension exists between designers' comfort with physical artifacts and the need for effective remote collaboration: physical objects live in one place. Previous research and technologies to support remote collaboration have focused on shared electronic media. Current technologies force distributed teams to choose between the physical tools they prefer and the electronic communication mechanisms available. We present Distributed Designers' Outpost, a remote collaboration system based on The Designers' Outpost, a collaborative web site design tool that employs physical Post-it notes as interaction primitives. We extended the system for synchronous remote collaboration and introduced two awareness mechanisms: transient ink input for gestures and a blue shadow of the remote collaborator for presence. We informally evaluated this system with six professional designers. Designers were excited by the prospect of physical remote collaboration but found some coordination challenges in the interaction with shared artifacts.

Our contextual inquiry into the practices of oral historians unearthed a curious incongruity. While oral historians consider interview recordings a central historical artifact, these recordings sit unused after a written transcript is produced. We hypothesized that this is largely because books are more usable than recordings. Therefore, we created Books with Voices: bar-code augmented paper transcripts enabling fast, random access to digital video interviews on a PDA. We present quantitative results of an evaluation of this tangible interface with 13 participants. They found this lightweight, structured access to original recordings to offer substantial benefits with minimal overhead. Oral historians found a level of emotion in the video not available in the printed transcript. The video also helped readers clarify the text and observe nonverbal cues.

To form a deep understanding of the present; we need to ?nd and engage history. We present an informal history capture and retrieval mechanism for collaborative, early-stage information design. This history system is implemented in the context of the Designers' Outpost, a wall-scale, tangible interface for collaborative web site design. The interface elements in this history system are designed to be ?uid and comfortable for early-phase design. As demonstrated by an informal lab study with six professional designers, this history system enhances the design process itself, and provides new opportunities for reasoning about the design of complex artifacts

In our previous studies into web design, we found that pens, paper, walls, and tables were often used for explaining, developing, and communicating ideas during the early phases of design. These wall-scale paper-based design practices inspired The Designers' Outpost, a tangible user interface that combines the affordances of paper and large physical workspaces with the advantages of electronic media to support information design. With Outpost, users collaboratively author web site information architectures on an electronic whiteboard using physical media (Post-it notes and images), structuring and annotating that information with electronic pens. This interaction is enabled by a touch-sensitive SMART Board augmented with a robust computer vision system, employing a rear-mounted video camera for capturing movement and a front-mounted high-resolution camera for capturing ink. We conducted a participatory design study with fifteen professional web designers. The study validated that Outpost supports information architecture work practice, and led to our adding support for fluid transitions to other tools.