Visual design instructors often provide multi-modal feedback, mixing annotations with text. Prior theory emphasizes the importance of actionable feedback, where “actionability” lies on a spectrum—from surfacing relevant design concepts to suggesting concrete fixes. How might creativity tools implement annotations that support such feedback, and how does the actionability of feedback impact novices’ process-related behaviors, perceptions of creativity, learning of design principles, and overall outcomes? We introduce VizCrit, a system for providing computational feedback that supports the actionability spectrum, realized through algorithmic issue detection and visual annotation generation. In a between-subjects study (N=36), novices revised a design under one of three conditions: textbook-based, awareness-centered, or solution-centered feedback. We found that solution-centered feedback led to fewer design issues and higher self-perceived creativity compared with textbook-based feedback, although expert ratings on creativity showed no significant differences. We discuss the implications for AI in Creativity Support Tools, including the potential of calibrating feedback actionability to help novices balance productivity with learning, growth, and developing design awareness.

Generative AI has enabled novice designers to quickly create professional-looking visual representations for product concepts. However, novices have limited domain knowledge that could constrain their ability to write prompts that effectively explore a product design space. To understand how experts explore and communicate about design spaces, we conducted a formative study with 12 experienced product designers and found that experts -- and their less-versed clients -- often use visual references to guide co-design discussions rather than written descriptions. These insights inspired DesignWeaver, an interface that helps novices generate prompts for a text-to-image model by surfacing key product design dimensions from generated images into a palette for quick selection. In a study with 52 novices, DesignWeaver enabled participants to craft longer prompts with more domain-specific vocabularies, resulting in more diverse, innovative product designs. However, the nuanced prompts heightened participants' expectations beyond what current text-to-image models could deliver. We discuss implications for AI-based product design support tools.

Voice-based social media platforms that enable attendees to have real-time, ephemeral interactions with each other—such as X-Spaces, Discord, and Clubhouse—have seen considerable growth in recent years. While prior research on these spaces has predominantly focused on moderating harms, our work seeks to understand emergent practices employed by hosts to proactively shape their discussion space— focusing on the facilitation aspect of moderation duties. Drawing on facilitation strategies, we study these practices through three comprehensive studies using mixed-methods: survey of social-audio users, co-design interviews, and analyzing training sessions for hosts. Our findings reveal insights into the issues faced by hosts and attendees, current facilitation practices, opinions on technological solutions, and factors that could be responsible for some of the identified issues such as the available training for hosts. We found that hosts themselves are often significant sources of issues due to practices such as focusing more on self-promotion than facilitating discussions. In addition, host training sessions seem to encourage behaviors that contribute to the negative perception of hosts. We draw on outcomes from co-design interviews to guide the design of future tools to support hosts in facilitating social-audio spaces. Our findings provide insights that could help create a more positive experience for both hosts and attendees.

Advances in AI have opened up the potential for creativity tools to computationally generate design feedback. In a future when designers can request feedback anytime on demand, how would the timing of these requests impact novices’ creative learning processes? What are the tradeoffs of providing access to feedback throughout a design task (in-action) versus only providing feedback after (on-action)? We explored these questions through a Wizard-of-Oz study (N=20) using an interactive design probe, where participants could request feedback either throughout the design process or only after they complete a full draft. We found that in-action participants frequently request feedback, resulting in better improvements as indicated by a greater decrease in issues in their final design. However, we saw that in-action feedback can also risk users overly relying on feedback instead of engaging in more holistic self-evaluation. We discuss the implications of our insights on designing tools for creative feedback.

What is the potential value and role for AI to facilitate real-time creative discussions? The paper explores principles for Generative-AI based conversational support by investigating how humans – playing the role of an AI agent – generate contextual conversational cues to guide an ideation session. We studied n=42 people (14 triads) brainstorming through a remote meeting design probe that allows a wizard facilitator to oversee the ideation and send text-based cues that appear real-time in the ideator interface. Thematic analysis of conversations, cues and post-hoc reflections by facilitators uncovered focal points, strategies and challenges. Notably, 44% of the cues sent out by the facilitators were either dismissed or ignored because they did not notice the cue update. When ideators did notice cues, certain facilitator strategies impacted the conversation more than others. Based on our analysis, we present design opportunities to improve generative AI-based systems to better support real-time creative collaborations.

As search evolves and Generative AI enables users to express more complex information needs and goals, it is an opportune moment to investigate how the search for information influences creativity. Little is known about how creators — especially novices who lack domain-specific terminology — use web search when developing an idea, and vice versa, how new information shapes an idea. To investigate how ideas evolve through web search, we conducted an online lab study with 56 design students who engaged in a 3-week product redesign project. Through a mixed-method analysis of web search logs, surveys, and interviews, we report on the different search behaviors, strategies, challenges and four distinct patterns–Orienters, Refiners, Confirmers, and Pivoters–that illustrate how the impact of search depends on the maturity of an idea. We discuss design opportunities to enhance web search systems for ideation and pedagogical interventions to teach creators how to improve idea-centric search.

Peer review is a cornerstone of science. Research communities conduct peer reviews to assess contributions and to improve the overall quality of science work. Every year, new community members are recruited as peer reviewers for the first time. How could technology help novices adhere to their community’s practices and standards for peer reviewing? To better understand peer review practices and challenges, we conducted a formative study with 10 novices and 10 experts. We found that many experts adopt a workflow of annotating, note-taking, and synthesizing notes into well-justified reviews that align with community standards. Novices lack timely guidance on how to read and assess submissions and how to structure paper reviews. To support the peer review process, we developed ReviewFlow – an AI-driven workflow that scaffolds novices with contextual reflections to critique and annotate submissions, in-situ knowledge support to assess novelty, and notes-to-outline synthesis to help align peer reviews with community expectations. In a within-subjects experiment, 16 inexperienced reviewers wrote reviews in two conditions: using ReviewFlow and using a baseline environment with minimal guidance. With ReviewFlow, participants produced more comprehensive reviews, identifying more pros and cons. However, they still struggled to provide actionable suggestions to address the weaknesses. While participants appreciated the streamlined process support from ReviewFlow, they also expressed concerns about using AI as part of the scientific review process. We discuss the implications of using AI to scaffold the peer review process on scientific work and beyond.

Advances in AI, particularly large language models (LLMs), can transform creative work. When developing a new idea, LLMs can help designers gather information, find competitors, and generate alternatives. However, LLM responses tend to be long-winded or contain inaccuracies, placing a burden on users to carefully synthesize information. In our formative studies with 52 students and five instructors, we find that novice designers typically lack guidance on how to compose prompts, reflect critically on LLM responses, and extract key information to help shape an idea. Building on these insights, we explore an alternative approach for interacting with LLMs, not via chat, but rather through structured templates. Collaborative design templates are a well-established strategy for helping novices think, organize information, and reflect on creative work. Developed as a digital whiteboard plugin, Jamplate integrates LLM capabilities into design templates, streamlining the collection and organization of user-generated content and LLM responses within the template structure. In a preliminary study with 8 novice designers, participants expressed that Jamplate’s reflective questions and in-situ guidance improved their ability to think critically and improve ideas more effectively. We discuss the potential of designing LLM-enhanced templates to instigate critical reflection.

Recent advances in Large Language Models (LLMs) show the potential to significantly augment or even replace complex human writing activities. However, for complex tasks where people need to make decisions as well as write a justification, the trade offs between making work efficient and hindering decisions remain unclear. In this paper, we explore this question in the context of designing intelligent scaffolding for writing meta-reviews for an academic peer review process. We prototyped a system called "MetaWriter'' trained on five years of open peer review data to support meta-reviewing. The system highlights common topics in the original peer reviews, extracts key points by each reviewer, and on request, provides a preliminary draft of a meta-review that can be further edited. To understand how novice and experienced meta-reviewers use MetaWriter, we conducted a within-subject study with 32 participants. Each participant wrote meta-reviews for two papers: one with and one without MetaWriter. We found that MetaWriter significantly expedited the authoring process and improved the coverage of meta-reviews, as rated by experts, compared to the baseline. While participants recognized the efficiency benefits, they raised concerns around trust, over-reliance, and agency. We also interviewed six paper authors to understand their opinions of using machine intelligence to support the peer review process and reported critical reflections. We discuss implications for future interactive AI writing tools to support complex synthesis work.

Traditional design galleries enable users to search for examples based on surface attributes (e.g., color or style), and largely obscure underlying principles (e.g., hierarchy or readability). We conducted three studies to explore how galleries could be constructed to help novices learn key design principles. Study 1 revealed that novices gain perspective by observing how designs evolve throughout a process. Study 2 found that novices are better at identifying design issues when viewing iterations that show improvements for just one principle at a time, rather than multiple. Building on these insights, we created ProcessGallery, a tool that enables users to browse contrasting pairs of early-and-late iterations of designs that highlight key improvements organized by design principles. In Study 3, a within-subjects experiment, sixteen participants iterated on a seed design after viewing examples in ProcessGallery versus a traditional gallery. Using ProcessGallery, participants found more appropriate examples, assessed designs better, and preferred ProcessGallery for learning compared to a traditional gallery.

Online whiteboards are becoming a popular way to facilitate collaborative design work, providing a free-form environment to curate ideas. However, as templates are increasingly being used to scaffold contributions from non-experts designers, it is crucial to understand their impact on the creative process. In this paper, we present the results from a study with 114 students in a large introductory design course. Our results confirm prior findings that templates benefit students by providing a starting point, a shared process, and the ability to access their own work from previous steps. While prior research has criticized templates for being too rigid, we discovered that using templates within a free-form environment resulted in visual patterns of free-form curation where concepts were spatially organized, clustered, color-coded, and connected using arrows and lines. We introduce the concept of ‘Free-form Templates’ to illustrate how templates and free-form curation can be synergistic.

Recent years have seen a rise in social virtual reality (VR) platforms that allow people to interact in real-time through voice and gestures. The ephemeral nature of communication on these platforms can enable new forms of harmful behavior and new challenges for moderators. We performed virtual field research on three VR environments (AltspaceVR, Horizon Worlds, Rec Room). Based on observing 100 scheduled events, our analysis uncovered 13 distinct types of potentially harmful behaviors enabled by real-time voice, embodied interactions, and platform affordances. We witnessed potential harm at 45% of our observed events; only 24% of these incidents were addressed by moderators. To understand moderation practices, we conducted interviews with 11 moderators to investigate how they assess real-time interactions and how they operate within the current state of moderation tools. Our work sheds light on how moderation tools and practices must evolve to meet the new challenges of social VR.

Large language models (LLMs) have recently soared in popularity due to their ease of access and the unprecedented ability to synthesize text responses to diverse user questions. However, LLMs like ChatGPT present significant limitations in supporting complex information tasks due to the insufficient affordances of the text-based medium and linear conversational structure. Through a formative study with ten participants, we found that LLM interfaces often present long-winded responses, making it difficult for people to quickly comprehend and interact flexibly with various pieces of information, particularly during more complex tasks. We present Graphologue, an interactive system that converts text-based responses from LLMs into graphical diagrams to facilitate information-seeking and question-answering tasks. Graphologue employs novel prompting strategies and interface designs to extract entities and relationships from LLM responses and constructs node-link diagrams in real-time. Further, users can interact with the diagrams to flexibly adjust the graphical presentation and to submit context-specific prompts to obtain more information. Utilizing diagrams, Graphologue enables graphical, non-linear dialogues between humans and LLMs, facilitating information exploration, organization, and comprehension.

We introduce the Idea Machine, a creativity support tool that leverages large language models (LLMs) to empower people engaged in idea generation tasks. The tool includes a number of affordances that can be used to enable various levels of automation and intelligent support. Each idea entered into the system can be expanded, rewritten, or combined with other ideas or concepts. An idea suggestion mode can also be enabled to make the system proactively suggest ideas.

Many AI system designers grapple with how best to collect human input for different types of training data. Online crowds provide a cheap on-demand source of intelligence, but they often lack the expertise required in many domains. Experts offer tacit knowledge and more nuanced input, but they are harder to recruit. To explore this trade off, we compared novices and experts in terms of performance and perceptions on human intelligence tasks in the context of designing a text-based conversational agent. We developed a preliminary chatbot that simulates conversations with someone seeking mental health advice to help educate volunteer listeners at 7cups.com. We then recruited experienced listeners (domain experts) and MTurk novice workers (crowd workers) to conduct tasks to improve the chatbot with different levels of complexity. Novice crowds perform comparably to experts on tasks that only require natural language understanding, such as correcting how the system classifies a user statement. For more generative tasks, like creating new lines of chatbot dialogue, the experts demonstrated higher quality, novelty, and emotion. We also uncovered a motivational gap: crowd workers enjoyed the interactive tasks, while experts found the work to be tedious and repetitive. We offer design considerations for allocating crowd workers and experts on input tasks for AI systems, and for better motivating experts to participate in low-level data work for AI.

Examples help students learn insights about key domain principles and processes. However, little is known about how students leverage the Web to discover and learn from examples. In a comparative study, seventy undergraduate students leveraged three types of platforms--- search-based, crit-based, and portfolio-based platforms---to find examples that represent contrasting cases of two design principles. Students reported how each platform's features and mechanisms affected their approach. We identify three main strategies students employed for finding examples on the Web: developing keywords, visually comparing multiple examples, and leveraging community feedback to assess example quality. Our results also indicate that, despite giving access to many examples, none of the existing platforms provide explicit support for learning. We distill three guidelines for creating learner-centered online design galleries to help future learners gain design knowledge.

Web search is increasingly used to satisfy complex, exploratory information goals. Exploring and synthesizing information into knowledge can be slow and cognitively demanding due to a disconnect between search tools and sense-making workspaces. Our work explores how we might integrate contextual query suggestions within a person’s sensemaking environment. We developed InterWeave a prototype that leverages a human wizard to generate contextual search guidance and to place the suggestions within the emergent structure of a searchers’ notes. To investigate how weaving suggestions into the sensemaking workspace affects a user’s search and sensemaking behavior, we ran a between-subjects study (n=34) where we compare InterWeave’s in context placement with a conventional list of query suggestions. InterWeave’s approach not only promoted active searching, information gathering and knowledge discovery, but also helped participants keep track of new suggestions and connect newly discovered information to existing knowledge, in comparison to presenting suggestions as a separate list. These results point to directions for future work to interweave contextual and natural search guidance into everyday work.

Misinformation runs rampant on social media and has been tied to adverse health behaviors such as vaccine hesitancy. Crowdsourcing can be a means to detect and impede the spread of misinformation online. However, past studies have not deeply examined the individual characteristics - such as cognitive factors and biases - that predict crowdworker accuracy at identifying misinformation. In our study (n = 265), Amazon Mechanical Turk (MTurk) workers and university students assessed the truthfulness and sentiment of COVID-19 related tweets as well as answered several surveys on personal characteristics. Results support the viability of crowdsourcing for assessing misinformation and content stance (i.e., sentiment) related to ongoing and politically-charged topics like the COVID-19 pandemic, however, alignment with experts depends on who is in the crowd. Specifically, we find that respondents with high Cognitive Reflection Test (CRT) scores, conscientiousness, and trust in medical scientists are more aligned with experts while respondents with high Need for Cognitive Closure (NFCC) and those who lean politically conservative are less aligned with experts. We see differences between recruitment platforms as well, as our data shows university students are on average more aligned with experts than MTurk workers, most likely due to overall differences in participant characteristics on each platform. Results offer transparency into how crowd composition affects misinformation and stance assessment and have implications on future crowd recruitment and filtering practices.

How does presenting comments in a news article affect the ways that readers engage with and retain information about news? This paper presents results from a controlled experiment investigating effects related to different strategies for promoting discussion at news websites (N=336 participants). The strategies include highlighting specific comments about a data visualization, providing prompts with the comments, and annotating prompts on the visualization. By comparison to a simple list of comments (baseline), our analysis found that annotations contributed to higher levels of participant engagement in the discussion, yet lower levels of knowledge retention related to the article. These findings raise new considerations about whether and how to integrate discussion content into news and points toward future content moderation systems that assist in representing and eliciting discussion at news websites.

Problem framing—the process of defining a problem—has been described by many researchers and designers as the crux of the design process. However, novice designers struggle with problem framing. To better understand this process and the potential for scaffolding, we conducted two studies. In the first study, we analyzed 41 problem statements from an introductory design course and found that novices often omit key information, like the primary stakeholder or the obstacles they face. To get novices to reflect on and include necessary design information, we created a tool called ProbLib that cues novices to explicitly reflect on aspects of the problem such as the stakeholders. To evaluate this approach, we conducted a between-subjects study (N=73) to compare ProbLib with an unstructured open text form. We found that participants using ProbLib wrote higher quality statements, included more information, and were more confident about specifying design needs. We observed creative behaviors such as brainstorming and analogical reasoning.

When exploring a new domain through web search, people often struggle to articulate queries because they lack domain-specific language and well-defined informational goals. Perhaps search tools rely too much on the query to understand what a searcher wants. Towards expanding this contextual understanding of a user during exploratory search, we introduce a novel system, CoNotate, which offers query suggestions based on analyzing the searcher’s notes and previous searches for patterns and gaps in information. To evaluate this approach, we conducted a within-subjects study where participants (n=38) conducted exploratory searches using a baseline system (standard web search) and the CoNotate system. The CoNotate approach helped searchers issue significantly more queries, and discover more terminology than standard web search. This work demonstrates how search can leverage user-generated content to help people get started when exploring complex, multi-faceted information spaces.

While research shows that web search plays a role throughout the creative process, less is known about about how people use web search to learn and frame their thinking about an open problem. People need web search to gather information about a problem area, but this can also influence the rest of the creative process. To understand how web search affects early-stage design, we collected and analyzed search log and self-report data from 34 students in a project-based design class. Participants reported struggling with scoping broad, ill-defined information goals into queries, learning domain-specific language, and assessing the usefulness of information. Analysis found that more active and diverse search behavior (i.e. issuing more frequent and diverse queries, and opening more webpages) related to more progress in early-stage design (i.e. gathering more facts, articulating more insights, and developing better problem frames). Based on these findings, we discuss implications for designing search tools to support peoples' creative processes.

Many organizations have adopted design processes that integrate community voices to discover the real problems that communities face. Online discussion forums offer a familiar and flexible technology that can help facilitate discussion around problems and potential solutions. However, we lack understanding about what information community members share, how that information is structured, and how social interactions affect design processes at scale. This paper presents a mixed-methods analysis of Canvas, a learning management system, which enables users to contribute to the design of the platform by sharing and deliberating on problems and solutions in a discussion forum. We collected and analyzed 1412 ideas and 18,335 associated comments shared on the Canvas discussion forum. We found that the distributed nature of design information, the presence of duplicate ideas, and contributors' gaming behaviors made it difficult for the community to make sense of the design discussion. These gaming behaviors also constitute a new concern for participatory design research. Finally, we reflect on how Canvas community members contribute information to a shared design space and how future systems could more effectively coordinate community design efforts.

Due to challenges around low-quality comments and misinformation, many news outlets have opted to turn off commenting features on their websites. The New York Times (NYT), on the other hand, has continued to scale up its online discussion resources to reach large audiences. Through interviews with the NYT moderation team, we present examples of how moderators manage the first ~24 hours of online discussion after a story breaks, while balancing concerns about journalistic credibility. We discuss how managing comments at the NYT is not merely a matter of content regulation, but can involve reporting from the "community beat" to recognize emerging topics and synthesize the multiple perspectives in a discussion to promote community. We discuss how other news organizations---including those lacking moderation resources---might appropriate the strategies and decisions offered by the NYT. Future research should investigate strategies to share and update the information generated about topics in the news through the course of content moderation.

Creating truly original ideas requires extensive knowledge of existing ideas. Navigating prior examples can help people to understand what has already been done and to assess the quality of their own ideas through comparison. The creativity literature has suggested that the conceptual distance between a proposed solution and a potential inspiration can influence one's thinking. However, less is known about how creators might use data about conceptual distance when exploring a large repository of ideas. To investigate this, we created a novel tool for exploring examples called IdeateRelate that visualizes 600+ COVID-related ideas, organized by their similarity to a new idea. In an experiment that compared the IdeateRelate visualization to a simple list of examples, we found that users in the Viz condition leveraged both semantic and categorical similarity, curated a more similar set of examples, and adopted more language from examples into their iterated ideas (without negatively affecting the overall novelty). We discuss implications for creating adaptive interfaces that provide creative inspiration in response to designers' ideas throughout an iterative design process.

Feedback is a key element of project-based learning, but only if students reflect on and learn from the feedback they receive. Students often struggle to deeply engage with feedback, whether due to lack of confidence, time, or skill. This work seeks to identify challenges that make reflecting on feedback difficult for students, and to design possible solutions for supporting reflection. Through observing two university game design courses, our research found that without concrete reflection strategies, students tended to be attracted to feedback that looks useful, but does not necessarily them move forward. When we introduced three different reflection scaffolds to support students, we found that the most effective approach promoted interactive learning by allowing time for self-reflection before team reflection, offering time limits, providing activities for feedback prioritization, helping teams align their goals, and equalizing team member participation. We present design guidelines for future systems to support reflection on feedback.

While most crowd work seeks consistent answers, creative domains often seek more diverse input. The typical crowd mechanisms for controlling quality may stifle creativity, yet removing them altogether could just produce noise. Schemas and metadata provide two mechanisms for embedding existing knowledge into task environments. Schemas are expert-derived patterns designed to structure how people think through a problem. Metadata, on the other hand, illustrate a range of creative input that fits within the structure of a schema. To understand the relative effects of schemas and metadata, we conducted a study where crowd workers are asked to generate creative interpretations for a set of placemaking examples. Crowd workers were guided either by schema plus metadata, schema alone, or neither. We found that showing schema along with crowd-produced metadata helped workers contribute interpretations that are both more on-topic and diverse, compared to using the schema alone or no schema. We discuss the implications on how crowds can creatively build on insights shared by others.

Creative workers frequently turn to online critique communities for feedback on their work. While past research has focused primarily on how to yield better feedback from providers, less is known about the strategies feedback seekers use to engage providers and request feedback. We present two studies to explore the feedback exchange dynamics between feedback requesters and providers in the subreddit community, r/design\_critiques. In Study 1, we interviewed 12 community members and found that while creators have strategies to request feedback, they expressed uncertainty about whether and how to include details about the design context, personal background, and specific feedback needs. In Study 2, through a mixed-method analysis, we identified how specific request strategies impact the quantity and quality of community feedback, and found several key, but undervalued strategies: signaling as a novice, critiquing one's own design, and providing design variants. These strategies led to better community response, but were rarely used. We offer design implications around how to leverage these insights to improve online feedback exchange

News websites can facilitate global discussions about civic issues, but the financial cost and burden of moderating these forums has forced many to disable their commenting systems. In this paper, we consider the role that data visualizations play in online discussion around a civic issue, through an analysis of how people talk about climate change data in the comment threads at three news websites (i.e., Breitbart news, the Guardian, the New York Times). We find that out of 6,525 comments, only 2.4% reference data visualizations in the articles. While rare, the paper presents illustrative examples of how people refer to data---their collection, analysis, and visual representation---to engage with an article's narrative. Using text classification techniques we identify several features related to the content of comments that contain data-centered talk, such as article cosine similarity, hyperlinks, and comparison terms. Finally, we discuss potential ways that newsrooms might apply this analysis to promote data literacy, data science, and to foster community around shared experiences.

Team dating, or small-group interactions, can expose people to diverse perspectives and inform the potential for longer-term collaboration. However, rapidly configuring groups and facilitating interactions among strangers can be difficult, especially in co-located settings. We present ProtoTeams, a system that leverages personal mobile devices to support rapid group formation, to facilitate group activities, and to collect data about the potential for future collaboration. We report on a field study where 406 students in eight different project-based classes used ProtoTeams to interact with classmates through multiple rounds of brief discussion activities before selecting teammates for a term project. We found that the system enables groups to form in about one minute, allows for meaningful interactions with a diverse range of peers, and can significantly influence subsequent teammate selection. We discuss design implications and challenges for in-person team dating in classrooms and other contexts.

Advancements in digital civics have enabled leaders to engage and gather input from a broader spectrum of the public. However, less is known about the analysis process around community input and the challenges faced by civic leaders as engagement practices scale up. To understand these challenges, we conducted 21 interviews with leaders on civic-oriented projects. We found that at a small-scale, civic leaders manage to facilitate sensemaking through collaborative or individual approaches. However, as civic leaders scale engagement practices to account for more diverse perspectives, making sense of the large quantity of qualitative data becomes a challenge. Civic leaders could benefit from training in qualitative data analysis and simple, scalable collaborative analysis tools that would help the community form a shared understanding. Drawing from these insights, we discuss opportunities for designing tools that could improve civic leaders' ability to utilize and reflect public input in decisions.

Exposing people to concepts created by others can inspire novel combinations of concepts, or conversely, lead people to simply emulate others. But how does the type of exposure affect creative outcomes in online collaboration where dyads interact for short tasks? In this paper, we study the creative outcomes of dyads working together online on a slogan writing task under different types of interactions: providing both the partner's idea and their explanation for that idea, enabling synchronous chat, and only exposing a person to their partner's idea without any explanation. We measure the creative outcome and define text-similarity-based metrics (e.g., mimicry, convergence, and fixation) to disentangle the interactions. The results show that having partners explain their ideas leads to largest improvement in creative outcome. In contrast, participants who chatted were more likely to reach convergence on their final slogans. Our work sheds lights on how different online interactions may create trade-offs in creative collaborations.

Peer feedback is essential for learning in project-based disciplines. However, students often need guidance when acting as either a feedback provider or a feedback receiver, both to gain from peer feedback and to criticize their peers' work. This paper explores how to more effectively scaffold this exchange such that peers more deeply engage in the feedback process. Within a game design course, we introduced different processes for feedback receivers to write questions to guide peer feedback. Feedback receivers wrote four main types of guiding questions: improve, share, brainstorm, critique. We found that "improve'' questions tended to lead to better feedback (more specific, critical, and actionable) than other question types, but feedback receivers wrote improve questions least often. We offer insights on how best to scaffold the question-writing process to facilitate peer feedback exchange.

Examples provide a source of inspiration for creating designs, but can they help improve the feedback process? Supplementing design feedback with examples could help recipients see issues clearly, identify concrete steps for improvement, and integrate novel ideas. Two online studies investigated how to support novices providing feedback on visual poster designs in an online context. Study One found that feedback providers select poster examples that complement their feedback and align with a provided rubric. Study Two shows that feedback providers give more specific, actionable, and novel input when using an example-centric approach, as opposed to text alone. To support this, we designed Paragon, an interface to efficiently browse examples using metadata. Finally, we discuss implications for collecting examples from the Web and structuring the design feedback process.

While urban design affects the public, most people do not have the time or expertise to participate in the process. Many online tools solicit public input, yet typically limit interaction to collecting complaints or early-stage ideas. This paper explores how to engage the public in more complex stages of urban design without requiring a significant time commitment. After observing workshops, we designed a system called CommunityCrit that offers micro-activities to engage communities in elaborating and evaluating urban design ideas. Through a four-week deployment, in partnership with a local planning group seeking to redesign a street intersection, CommunityCrit yielded 352 contributions (around 10 minutes per participant). The planning group reported that CommunityCrit provided insights on public perspectives and raised awareness for their project, but noted the importance of setting expectations for the process. People appreciated that the system provided a window into the planning process, empowered them to contribute, and supported diverse levels of skills and availability.

From Fortune 500 companies to local communities, organizations often strive to build a shared understanding about complex problems. Design competitions provide a compelling approach to create incentives and infrastructure for gathering insights about a problem-space. In this paper, we present an analysis of a two-month civic design competition focused on transportation challenges in a major US city. We examine how the event structure, discussion platform, and participant interactions affected how a community collectively discussed design constraints and proposals. Ninety-two participants took part in the competition's online discussion, hosted on Slack. Applying a mixed-methods analysis, we found that participants shared less as they settled into teams and, due to the discussion system, had difficulty seeing how topics connected across channels; we also learned that certain messages led participants to add depth to existing topics. Based on the findings we provide recommendations for civic competitions aimed at building knowledge around a problem.

Online crowds can help infuse creativity into the design process, but traditional strategies for leveraging them, such as large-scale ideation platforms, require time and organizational effort in order to obtain results. We propose a new method for crowd-based ideation that simplifies the process by having smaller crowds join in-person ideators during synchronous creative sessions. Our system Crowdboard allows online crowds to provide real-time creative input during early-stage design activities, such as brainstorming or concept mapping. The system enables in-person ideators to develop ideas on a physical or digital whiteboard which is augmented with real-time creative input from online participants who see and hear a live broadcast of the meeting. We validate Crowdboard via two user studies in which dyads of in-person ideators brainstormed with the help of crowd ideators. Our studies suggest that Crowdboard can effectively enhance ongoing brainstorming sessions, but also revealed key challenges for how to better facilitate interactions among in-person and crowd ideators.

Collaborative ideation systems can help people generate more creative ideas by exposing them to ideas different from their own. However, there are competing theoretical views on whether and when such exposure is helpful. Associationist theory suggests that exposing ideators to ideas that are semantically far from their own maximizes novel combinations of ideas. In contrast, SIAM theory cautions that systems should offer far ideas only when ideators reach an impasse (a cognitive state in which they have exhausted ideas within a particular category), and offer near ideas during productive ideation (a cognitive state in which they are actively exploring ideas within a category), which maximizes exploration within categories. Our research compares these theoretical recommendations. In an online experiment, 245 participants generated ideas for a themed wedding; we detected and validated participants' cognitive states using a combination of behavioral and neuroimaging data. Receiving far ideas during productive ideation resulted in slower ideation and less within-category exploration, without significant benefits for novelty, compared to receiving no inspirations. Participants were also more likely to hit an impasse when receiving far ideas during productive ideation. These findings suggest that far inspirational ideas can harm creativity if received during productive ideation.

Feedback from diverse audiences can contain ambiguity and contradictions, making it difficult to interpret and act on. To promote deeper interpretation of feedback, we tested the effects of combining a reflection activity and reviewing external feedback for an iterative design task. Designers (N=90) created a design and revised it after a) performing a reflection activity before reviewing feedback, b) performing the reflection after reviewing feedback, c) performing the reflection only, or d) reviewing the feedback only. We measured design quality, depth of revision, perceived effort, and confidence; and categorized the content produced from the reflections. We found that performing reflection after feedback review led to the largest increase in perceived quality for the revised designs, and performing reflection and feedback review regardless of the order resulted in the most extensive revision. Our results also showed that performing the reflection alone yielded outcomes that were similar to when only reviewing feedback, and either activity led to better outcomes than the control condition (no feedback or reflection). Designers stated that the reflection helped them recall their goals, question their choices, and prioritize revisions. We argue that designers should perform a lightweight, explicit reflection to enhance their iterative process, and discuss implications for feedback platforms.

To meet the demand for authentic, timely, and affordable feedback, researchers have explored technologies to connect designers with feedback providers online. While researchers have implemented mechanisms to improve the content of feedback, most systems for online feedback exchange do not support an end-to-end cycle, from help-seeking to sense-making to action. Building on extant literature in learning sciences, design, organizational behavior, and online communities, we propose a conceptual framework to highlight critical processes that affect online feedback exchange. We contribute research questions for future feedback systems and argue that online feedback systems must be able to support designers through five activities that happen before, during, and after the feedback exchange. Our framework suggests that systems should address broader socio-psychological factors, such as how intent should be communicated online, how dialogue can support the interpretation of feedback, and how to balance the tradeoffs of anonymizing feedback providers.

Designers are increasingly leveraging online crowds; yet, online contributors may lack the expertise, context, and sensitivity to provide effective critique. Rubrics help feedback providers but require domain experts to write them and may not generalize across design domains. This paper introduces and tests a novel semi-automated method to support feedback providers by analyzing feedback language. In our first study, 52 students from two design courses created design solutions and received feedback from 176 online providers. Instructors, students, and crowd contributors rated the helpfulness of each feedback response. From this data, an algorithm extracted a set of natural language features (e.g., specificity, sentiment etc.) that correlated with the ratings. The features accurately predicted the ratings and remained stable across different raters and design solutions. Based on these features, we produced a critique style guide with feedback examples - automatically selected for each feature - to help providers revise their feedback through self-assessment. In a second study, we tested the validity of the guide through a between-subjects experiment (n=50). Providers wrote feedback on design solutions with or without the guide. Providers generated feedback with higher perceived helpfulness when using our style-based guidance.

Peer feedback is a central activity for project-based design education. The prevalence of devices carried by students and the emergence of novel peer feedback systems enables the possibility of collecting and sharing feedback immediately between students during class. However, pen and paper is thought to be more familiar, less distracting for students, and easier for instructors to implement and manage. To evaluate the efficacy of in-class digital feedback systems, we conducted a within-subjects study with 73 students during two weeks of a game design course. After short student presentations, while instructors provided verbal feedback, peers provided feedback either on paper or through a device. The study found that both methods yielded comments of similar quality and quantity, but the digital approach provided additional ways for students to participate and required less effort from the instructors. While both methods produced similar behaviors, students held inaccurate perceptions about their behavior with each method. We discuss design implications for technologies to support in-class feedback exchange.

As demand for design education increases, instructors are struggling to provide timely, personalized feedback for student projects. Gathering feedback from classroom peers and external crowds offer scalable approaches, but there is little evidence of how they compare. We report on a study in which students (n=127) created early- and late-stage prototypes as part of nine-week projects. At each stage, students received feedback from peers and external crowds: their own social networks, online communities, and a task market. We measured the quality, quantity and valence of the feedback and the actions taken on it, and categorized its content using a taxonomy of critique discourse. The study found that peers produced feedback that was of higher perceived quality, acted upon more, and longer compared to the crowds. However, crowd feedback was found to be a viable supplement to peer feedback and students preferred it for projects targeting specialized audiences. Feedback from all sources spanned only a subset of the critique categories. Instructors may fill this gap by further scaffolding feedback generation. The study contributes insights for how to best utilize different feedback sources in project-based courses.

Forming work teams involves matching people with complementary skills and personalities, but requires obtaining such data a priori. We introduce team dating, where people interact on brief tasks before working with a dedicated partner for longer, more complex tasks. We studied team dating through two online experiments. In Experiment 1, workers from a crowd platform independently wrote an ad slogan, discussed it with three consecutive people and evaluated their team date interactions. They then selected preferred teammates from a list showing average ratings for people they had dated and not dated. Results show that participants evaluated their dates based on evidence beyond externally judged slogan quality, and relied heavily on their dyad-specific judgments in selecting teammates. In Experiment 2, we replicated the individual and team dating tasks, and formed teams, either i) by honoring pairwise team dating preferences, ii) randomly from their pool of dates, or iii) randomly from those not dated. Results show that teams formed from preferred dates performed better on a final creative task compared to random dates or non-dates. Team dating provides a dynamic technique for forming ad hoc teams accounting for interpersonal dynamics. The initial interactions provided information that helped people select and work with an appropriate teammate.

Feedback is information that can improve task performance. Online communities, educational forums, and crowd-based feedback platforms all support feedback exchange among a more diverse set of sources than ever before, with greater control over how to moderate this exchange. In this work, we study how the power relationship between the source and receiver and the tone of language influence the recep-tivity, effort, and work performance resulting from online feedback exchange. We conducted an online experiment manipulating affective language and source of feedback on a writing task. We found that critiques with positive affec-tive language increased positive emotions and reduced participants' annoyance and frustration, which led to an increase in work quality, compared to critiques without positive language. Feedback without positive affective language led to more edits, but not better work outcomes. Participants reacted more positively to feedback from an anonymous source than from a peer or an authority. Our findings provide design implications for platforms to support more fruitful feedback exchange.

Team-based learning is a structured, small-group learning method that has been associated with many positive outcomes in traditional classroom settings. However, relatively little research has focused on how to form and support teams within online learning platforms, such as Massive Open Online Courses (MOOCs). A number of challenges arise for team formation in voluntary online classes: students may drop out and leave their team, and even if they do persist with the course, the team may not work together effectively. In this paper, we introduce a team-formation strategy that incorporates a deliberation process, where participants hold discussions in preparation for the collaboration task. First, we present a crowdsourced experiment that compares teams that are formed before or after a community deliberation process. Results demonstrate that teams engaging in a larger community deliberative process prior to team formation exhibit better team performance--as measured by team collaboration product quality--than pre-discussion teams. In a second crowdsourced experiment, we further explore the benefits of community-wide processes by automatically assigning teams based on participants' transactive interaction during deliberation. The results demonstrate advantages in terms of team performance for teams formed based on observed interactions during the community-level deliberation, compared to randomly formed teams. Finally, in a case study, we demonstrate how we successfully adapted the team formation strategy for use in a small MOOC.

Peer feedback systems enable students to get feedback without substantially burdening the instructor. However, current systems typically ask students to provide feedback after class; this introduces challenges for ensuring relevant, timely, diverse, and sufficient amounts of feedback, and reduces time available for student reflection. This paper explores the current landscape of peer feedback tools and introduces a novel system for in-class peer review called PeerPresents where students can quickly exchange feed-back on projects without being burdened by additional work outside of class. Through an exploratory study with Google docs and a preliminary evaluation of PeerPresents, we find students can receive immediate, copious, and diverse peer feedback through a structured in-class activity. Students also described the feedback they received as helpful and reported that they gave more feedback than without using the system. These early results demonstrate the potential benefits of in-class peer feedback systems.

Increasingly, designers seek feedback on their designs from crowd platforms such as social networks, Web forums, and paid task markets which demand different amounts of social capital, financial resources, and time. Yet it is unknown how the choice of crowd platform affects feedback generation. We conducted an online study where designers created initial designs and revised the designs based on crowd feedback. We measured the quantity, quality, and content of the feedback received at two iterations and from crowds driven by social status, enjoyment, and financial gain. Our results show, for example, that task markets yield more suggestions, online forums provide more process feedback, and social networks give the most suggestions without payment. We contribute an emergent framework for crowd feedback selection, opportunities for enhancing feedback services, and an experimental platform that researchers can adapt to reduce the burden of conducting online studies of design feedback.

Large-scale idea generation platforms often expose ideators to previous ideas. However, research suggests people generate better ideas if they see abstracted solution paths (e.g., descriptions of solution approaches generated through human sensemaking) rather than being inundated with all prior ideas. Automated and semi-automated methods can also offer interpretations of earlier ideas. To benefit from sensemaking in practice with limited resources, ideation platform developers need to weigh the cost-quality tradeoffs of different methods for surfacing solution paths. To explore this, we conducted an online study where 245 participants generated ideas for two problems in one of five conditions: 1) no stimuli, 2) exposure to all prior ideas, or solution paths extracted from prior ideas using 3) a fully automated workflow, 4) a hybrid human-machine approach, and 5) a fully manual approach. Contrary to expectations, human-generated paths did not improve ideation (as meas-ured by fluency and breadth of ideation) over simply showing all ideas. Machine-generated paths sometimes significantly improved fluency and breadth of ideation over no ideas (although at some cost to idea quality). These findings suggest that automated sensemaking can improve idea generation, but we need more research to understand the value of human sensemaking for crowd ideation.

The physical constraints of smartwatches limit the range and complexity of tasks that can be completed. Despite interface improvements on smartwatches, the promise of enabling productive work remains largely unrealized. This paper presents WearWrite, a system that enables users to write documents from their smartwatches by leveraging a crowd to help translate their ideas into text. WearWrite users dictate tasks, respond to questions, and receive notifications of major edits on their watch. Using a dynamic task queue, the crowd receives tasks issued by the watch user and generic tasks from the system. In a week-long study with seven smartwatch users supported by approximately 29 crowd workers each, we validate that it is possible to manage the crowd writing process from a watch. Watch users captured new ideas as they came to mind and managed a crowd during spare moments while going about their daily routine. WearWrite represents a new approach to getting work done from wearables using the crowd.

Online crowds are a promising source of new innovations. However, crowd innovation quality does not always match its quantity. In this paper, we explore how to improve crowd innovation with real-time expert guidance. One approach would for experts to provide personalized feed-back, but this scales poorly, and may lead to premature convergence during creative work. Drawing on strategies for facilitating face-to-face brainstorms, we introduce a crowd ideation system where experts monitor incoming ideas through a dashboard and offer high-level “inspirations” to guide ideation. A series of controlled experiments show that experienced facilitators increased the quantity and creativity of workers' ideas compared to unfacilitated workers, while Novice facilitators reduced workers' creativity. Analyses of inspiration strategies suggest these opposing results stem from differential use of successful inspiration strategies (e.g., provoking mental simulations). The results show that expert facilitation can significantly improve crowd innovation, but inexperienced facilitators may need scaffolding to be successful.

When personalities clash, teams operate less effectively. Personality differences affect face-to-face collaboration and may lower trust in virtual teams. For relatively short-lived assignments, like those of online crowdsourcing, personality matching could provide a simple, scalable strategy for effective team formation. However, it is not clear how (or if) personality differences affect teamwork in this novel context where the workforce is more transient and diverse. This study examines how personality compatibility in crowd teams affects performance and individual perceptions. Using the DISC personality test, we composed 14 five-person teams (N=70) with either a harmonious coverage of personalities (balanced) or a surplus of leader-type personalities (imbalanced). Results show that balancing for personality leads to significantly better performance on a collaborative task. Balanced teams exhibited less conflict and their members reported higher levels of satisfaction and acceptance. This work demonstrates a simple personality matching strategy for forming more effective teams in crowdsourcing contexts.

Expert feedback is valuable but hard to obtain for many designers. Online crowds can provide fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted an experiment with a 2x2 factorial design. Student designers received feedback on a visual design from both experts and novices, who produced feedback using either an expert rubric or no rubric. We found that rubrics helped novice workers provide feedback that was rated nearly as valuable as expert feedback. A follow-up analysis on writing style showed that student designers found feedback most helpful when it was emotionally positive and specific, and that a rubric increased the occurrence of these characteristics in feedback. The analysis also found that expertise correlated with longer critiques, but not the other favorable characteristics. An informal evaluation indicates that experts may instead have produced value by providing clearer justifications.

Prior work on creativity support tools demonstrates how a computational semantic model of a solution space can enable interventions that substantially improve the number, quality and diversity of ideas. However, automated semantic modeling often falls short when people contribute short text snippets or sketches. Innovation platforms can employ humans to provide semantic judgments to construct a semantic model, but this relies on external workers completing a large number of tedious micro tasks. This requirement threatens both accuracy (external workers may lack expertise and context to make accurate semantic judgments) and scalability (external workers are costly). In this paper, we introduce IdeaHound, an ideation system that seamlessly integrates the task of defining semantic relationships among ideas into the primary task of idea generation. The system combines implicit human actions with machine learning to create a computational semantic model of the emerging solution space. The integrated nature of these judgments allows IDEAHOUND to leverage the expertise and efforts of participants who are already motivated to contribute to idea generation, overcoming the issues of scalability inherent to existing approaches. Our results show that participants were equally willing to use (and just as productive using) IDEAHOUND compared to a conventional platform that did not require organizing ideas. Our integrated crowdsourcing approach also creates a more accurate semantic model than an existing crowdsourced approach (performed by external crowds). We demonstrate how this model enables helpful creative interventions: providing diverse inspirational examples, providing similar ideas for a given idea and providing a visual overview of the solution space.

Emerging online ideation platforms with thousands of example ideas provide an important resource for creative production. But how can ideators best use these examples to create new innovations? Recent work has suggested that not just the choice of examples, but also the timing of their delivery can impact creative outcomes. Building on existing cognitive theories of creative insight, we hypothesize that people are likely to benefit from examples when they run out of ideas. We explore two example delivery mechanisms that test this hypothesis: 1) a system that proactively provides examples when a user appears to have run out of ideas, and 2) a system that provides examples when a user explicitly requests them. Our online experiment (N=97) compared these two mechanisms against two baselines: providing no examples and automatically showing examples at a regular interval. Participants who requested examples themselves generated ideas that were rated the most novel by external evaluators. Participants who received ideas automatically when they appeared to be stuck produced the most ideas. Importantly, participants who received examples at a regular interval generated fewer ideas than participants who received no examples, suggesting that mere access to examples is not sufficient for creative inspiration. These results emphasize the importance of the timing of example delivery. Insights from this study can inform the design of collective ideation support systems that help people generate many high quality ideas.

Coding behavioral video is an important method used by researchers to understand social phenomenon. Unfortunately, traditional hand-coding approaches can take days or weeks of time to complete. Recent work has shown that these tasks can be completed quickly by leveraging the parallelism of large online crowds, but using the crowd introduces new concerns about accuracy, reliability, privacy, and cost. To explore these issues, we conducted interviews with 12 researchers who frequently code behavioral video, to investigate common practices and challenges with video coding. We find accuracy and privacy to be the researchers' primary concerns. To explore this more concretely, we used sample videos to investigate whether crowds can accurately recognize instances of commonly coded behaviors, and show that the crowd yields accurate results. Then, we demonstrate a method for obfuscating participant identity with a video blur filter, and find, as expected, that workers' ability to identify participants decreases as blur level increases. The workers' ability to accurately and reliably code behaviors also decreases, but not as steeply as the identity test. This trade-off between coding quality and privacy protection suggests that researchers can use online crowds to code for some key behaviors in video without compromising participant identity. We conclude with a discussion of how researchers can balance privacy and accuracy on their own data using a system we introduce called Incognito.

Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a web-based system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer's experience and benefits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services.

Online collaboration tools enable developers of interactive systems to quickly reach potential users for usability testing. Can these technologies serve designers who seek feedback on user needs during the earliest stages of design? Online needfinding may help designers create products and services that can target a more diverse user population. To explore this, we conducted a feasibility study to compare face-to-face methods with online needfinding sessions. We found that video can sufficiently capture nuanced reactions to preliminary concept storyboards, but that feedback providers need guidance and structure. We then introduce a tool for collecting early-stage design feedback from online participants and conduct a case study with a professional design team. The team conducted needfinding activities with local participants, as well as a cost-equivalent number of online participants The case study demonstrates that combining online crowdsourcing with a video survey tool provides a simple and cost-efficient way to collect early-stage feedback.

A growing number of large collaborative idea generation platforms promise that by generating ideas together, people can create better ideas than any would have alone. But how might these platforms best leverage the number and diversity of contributors to help each contributor generate even better ideas? Prior research suggests that seeing particularly creative or diverse ideas from others can inspire you, but few scalable mechanisms exist to assess diversity. We contribute a new scalable crowd-powered method for evaluating the diversity of sets of ideas. The method relies on similarity comparisons (is idea A more similar to B or C) generated by non-experts to create an abstract spatial idea map. Our validation study reveals that human raters agree with the estimates of dissimilarity derived from our idea map as much or more than they agree with each other. People seeing the diverse sets of examples from our idea map generate more diverse ideas than those seeing randomly selected examples. Our results also corroborate findings from prior research showing that people presented with creative examples generated more creative ideas than those who saw a set of random examples. We see this work as a step toward building more effective online systems for supporting large scale collective ideation.

Crowd feedback systems offer designers an emerging approach for improving their designs, but there is little empirical evidence of the benefit of these systems. This paper reports the results of a study of using a crowd feedback system to iterate on visual designs. Users in an introductory visual design course created initial designs satisfying a design brief and received crowd feedback on the designs. Users revised the designs and the system was used to generate feedback again. This format enabled us to detect the changes between the initial and revised designs and how the feedback related to those changes. Further, we analyzed the value of crowd feedback by comparing it with expert evaluation and feedback generated via free-form prompts. Results showed that the crowd feedback system prompted deep and cosmetic changes and led to improved designs, the crowd recognized the design improvements, and structured workflows generated more interpretative, diverse and critical feedback than free-form prompts.

Student entrepreneurs struggle to collect feedback on their product pitches in a classroom setting due to a lack of time, money, and access to motivated feedback providers. Online social networks present a unique opportunity for entrepreneurial students to quickly access feedback providers by leveraging their online social capital. In order to better understand how to improve crowdsourced online pitch feedback, we perform an experiment to test the effect of online anonymity on pitch feedback quality and quantity. We also test a communal feedback method—evenly distributing between teams feedback providers from the class’s collective online social networks—which would help all teams benefit from a useful amount of feedback rather than having some teams receive much more feedback than others. We found that feedback providers in the anonymous condition provided significantly more specific criticism and specific praise, which students rated as more useful. Furthermore, we found that the communal feedback method helped all teams receive sufficient feedback to edit their pitches. This research contributes an empirical investigation to the crowdsourcing community of how crowds through online social networks can help student entrepreneurs obtain authentic feedback to improve their work.

Learning about a new area of knowledge is challenging for novices partly because they are not yet aware of which topics are most important. The Internet contains a wealth of information for learning the underlying structure of a domain, but relevant sources often have diverse structures and emphases, making it hard to discern what is widely considered essential knowledge vs. what is idiosyncratic. Crowdsourcing offers a potential solution because humans are skilled at evaluating high-level structure, but most crowd micro-tasks provide limited context and time. To address these challenges, we present Crowdlines, a system that uses crowdsourcing to help people synthesize diverse online information. Crowdworkers make connections across sources to produce a rich outline that surfaces diverse perspectives within important topics. We evaluate Crowdlines with two experiments. The first experiment shows that a high context, low structure interface helps crowdworkers perform faster, higher quality synthesis, while the second experiment shows that a tournament-style (parallelized) crowd workflow produces faster, higher quality, more diverse outlines than a linear (serial/iterative) workflow.

User-centered designers often seek to synthesize data from user research into insights and a shared point of view among team members. This paper explores the synthesis process and opportunities for providing computational support. First, we present interviews with novice and expert designers on the common practices and challenges of syn-thesis. Based on these interviews, we developed digital whiteboard software support for sorting individual seg-ments of user research. The system separates out individual and group activity and helps the team externalize and syn-thesize their different views of the data. Through a case study, we explore two computer-supported approaches: a structured condition that externalizes the different perspec-tives on the data of each team member and an unstructured condition that allows each member to organize data into clusters. Novice designers tended to prefer the structured synthesis process, while more experienced designers pre-ferred to freely arrange information segments and create clusters on their own. We provide implications for design education and support tools for user research synthesis.

By definition, human-centered design relies on interaction with users. While interacting with users within industry can be challenging, fostering these interactions in a classroom setting can be even more difficult. This qualitative study explores the use of crowd-based design activities as a way to support student-user interactions online. We motivate these online methods through a survey of 27 design instructors, who identified common challenges of conducting student-user interactions in physical settings, including coordination constraints and geographical barriers to meeting in person. We then describe our research through design to create and test 10 activities in a classroom setting, including using Twitter for needfinding and using Reddit to brainstorm ideas with experts. Finally, we present an emergent framework outlining the design space for crowd-based design activities where students learn to use input from online crowds to inform their design work. We discuss plans to refine and expand the current set of activities for open access to instructors.

Organizing conference sessions around themes improves the experience for attendees. However, the session creation process can be difficult and time-consuming due to the amount of expertise and effort required to consider alternative paper groupings. We present a collaborative web application called Frenzy to draw on the efforts and knowledge of an entire program committee. Frenzy comprises (a) interfaces to support large numbers of experts working collectively to create sessions, and (b) a two-stage process that decomposes the session-creation problem into meta-data elicitation and global constraint satisfaction. Meta-data elicitation involves a large group of experts working simultaneously, while global constraint satisfaction involves a smaller group that uses the meta-data to form sessions.

A central tenet of HCI is that technology should be user-centric, with designs being based around social science findings about users. Nevertheless a repeated but critical challenge in design is translating empirical findings into actionable ideas that inform design, or generating implications for design. Despite various design methods aiming to bridge this gap, knowledge informing design is still seen as problematic. However there has been little empirical exploration into what design researchers understand by such design knowledge, the functions and principles behind their creation. We report on interviews with twelve expert HCI design researchers probing the roles and types of design implications, and the process of generating and evaluating them. We synthesize different types of design implications into a framework to guide their generation. Our findings identify a broader range than previously described, additional sources and heuristics supporting their development as well some important evaluation criteria. We discuss the value of these findings for interaction design research.

Analysts synthesize complex, qualitative data to uncover themes and concepts, but the process is time-consuming, cognitively taxing, and automated techniques show mixed success. Crowdsourcing could help this process through on-demand harnessing of flexible and powerful human cognition, but incurs other challenges including limited attention and expertise. Further, text data can be complex, high-dimensional, and ill-structured. We address two major challenges unsolved in prior crowd clustering work: scaffolding expertise for novice crowd workers, and creating consistent and accurate categories when each worker only sees a small portion of the data. To address these challenges we present an empirical study of a two-stage approach to enable crowds to create an accurate and useful overview of a dataset: A) we draw on cognitive theory to assess how re-representing data can shorten and focus the data on salient dimensions; and B) introduce an iterative clustering approach that provides workers a global overview of data. We demonstrate a classification-plus-context approach elicits the most accurate categories at the most useful level of abstraction.

In modern crowdsourcing markets, requesters face the challenge of training and managing large transient workforces. Requesters can hire peer workers to review others' work, but the value may be marginal, especially if the reviewers lack requisite knowledge. Our research explores if and how workers learn and improve their performance in a task domain by serving as peer reviewers. Further, we investigate whether peer reviewing may be more effective in teams where the reviewers can reach consensus through discussion. An online between-subjects experiment compares the trade-offs of reviewing versus producing work using three different organization strategies: working individually, working as an interactive team, and aggregating individuals into nominal groups. The results show that workers who review others' work perform better on subsequent tasks than workers who just produce. We also find that interactive reviewer teams outperform individual reviewers on all quality measures. However, aggregating individual reviewers into nominal groups produces better quality assessments than interactive teams, except in task domains where discussion helps overcome individual misconceptions.

Constructing a good conference schedule for a large multi-track conference needs to take into account the preferences and constraints of organizers, authors, and attendees. Creating a schedule which has fewer conflicts for authors and attendees, and thematically coherent sessions is a challenging task. Cobi introduced an alternative approach to conference scheduling by engaging the community to play an active role in the planning process. The current Cobi pipeline consists of committee-sourcing and author-sourcing to plan a conference schedule. We further explore the design space of community-sourcing by introducing attendee-sourcing -- a process that collects input from conference attendees and encodes them as preferences and constraints for creating sessions and schedule. For CHI 2014, a large multi-track conference in human-computer interaction with more than 3,000 attendees and 1,000 authors, we collected attendees’ preferences by making available all the accepted papers at the conference on a paper recommendation tool we built called Confer, for a period of 45 days before announcing the conference program (sessions and schedule). We compare the preferences marked on Confer with the preferences collected from Cobi’s author-sourcing approach. We show that attendee-sourcing can provide insights beyond what can be discovered by author-sourcing. For CHI 2014, the results show value in the method and attendees’ participation. It produces data that provides more alternatives in scheduling and complements data collected from other methods for creating coherent sessions and reducing conflicts.

Many crowd ideation systems seek to gather scores of ideas from people online. However, this often leads to many bad ideas and duplication. A dedicated facilitator who guides exploration of the solution space is a common and effective strategy for optimizing ideation in face-to-face brainstorming, but has not yet been explored in computer-supported crowd ideation. We introduce IdeaGens, a social ideation system for guided crowd brainstorming. IdeaGens divides the crowd into ideation and synthesis tasks, and enables efficient data-driven facilitation of the crowd’s ideation. This work can inform general strategies for shepherding the crowd to produce better results for complex collaborative tasks.

Behavioral researchers code video to extract systematic meaning from subtle human actions and emotions. While this has traditionally been done by analysts within a research group, recent methods have leveraged online crowds to massively parallelize this task and reduce the time required from days to seconds. However, using the crowd to code video increases the risk that private information will be disclosed because workers who have not been vetted will view the video data in order to code it. In this Work-in-Progress, we discuss techniques for maintaining privacy when using Glance to code video and present initial experimental evidence to support them.

Behavioral researchers spend considerable amount of time coding video data to systematically extract meaning from subtle human actions and emotions. In this paper, we present Glance, a tool that allows researchers to rapidly query, sample, and analyze large video datasets for behavioral events that are hard to detect automatically. Glance takes advantage of the parallelism available in paid online crowds to interpret natural language queries and then aggregates responses in a summary view of the video data. Glance provides analysts with rapid responses when initially exploring a dataset, and reliable codings when refining an analysis. Our experiments show that Glance can code nearly 50 minutes of video in 5 minutes by recruiting over 60 workers simultaneously, and can get initial feedback to analysts in under 10 seconds for most clips. We present and compare new methods for accurately aggregating the input of multiple workers marking the spans of events in video data, and for measuring the quality of their coding in real-time before a baseline is established by measuring the variance between workers. Glance's rapid responses to natural language queries, feedback regarding question ambiguity and anomalies in the data, and ability to build on prior context in followup queries allow users to have a conversation-like interaction with their data - opening up new possibilities for naturally exploring video data.

Industry relies on higher education to prepare students for careers in innovation. Fulfilling this obligation is especially difficult in classroom settings, which often lack authentic interaction with the outside world. Online crowdsourcing has the potential to change this. Our research explores if and how online crowds can support student learning in the classroom. We explore how scalable, diverse, immediate (and often ambiguous and conflicting) input from online crowds affects student learning and motivation for project-based innovation work. In a pilot study with three classrooms, we explore interactions with the crowd at four key stages of the innovation process: needfinding, ideating, testing, and pitching. Students reported that online crowds helped them quickly and inexpensively identify needs and uncover issues with early-stage prototypes, although they favored face-to-face interactions for more contextual feed-back. We share early evidence and discuss implications for creating a socio-technical infrastructure to more effectively use crowdsourcing in education.

In online creative communities, members work together to produce music, movies, games, and other cultural products. Despite the proliferation of collaboration in these communities, we know little about how these teams form and what leads to their ultimate success. Building on theories of social identity and exchange, we present an exploratory study of an online songwriting community. We analyze four years of longitudinal behavioral data using a novel path-based regression model that accurately predicts and reveals key variables about collab formation. Combined with a large-scale survey of members, we find that communication, nuanced complementary interest and status, and a balanced effort from both parties contribute to successful collaborations. We also discuss several applications of these findings for socio-technical infrastructures that support online creative production.

Creating sessions of related papers for a large conference is a complex and time-consuming task. Traditionally, a few conference organizers group papers into sessions manually. Organizers often fail to capture the affinities between papers beyond created sessions, making incoherent sessions difficult to fix and alternative groupings hard to discover. This paper proposes committeesourcing and authorsourcing approaches to session creation (a specific instance of clustering and constraint satisfaction) that tap into the expertise and interest of committee members and authors for identifying paper affinities. During the planning of ACM CHI'13, a large conference on human-computer interaction, we recruited committee members to group papers using two online distributed clustering methods. To refine these paper affinities — and to evaluate the committeesourcing methods against existing manual and automated approaches — we recruited authors to identify papers that fit well in a session with their own. Results show that authors found papers grouped by the distributed clustering methods to be as relevant as, or more relevant than, papers suggested through the existing in-person meeting. Results also demonstrate that communitysourced results capture affinities beyond sessions and provide flexibility during scheduling.

Effectively planning a large multi-track conference requires an understanding of the preferences and constraints of organizers, authors, and attendees. Traditionally, the onus of scheduling the program falls on a few dedicated organizers. Resolving conflicts becomes difficult due to the size and complexity of the schedule and the lack of insight into community members' needs and desires. Cobi presents an alternative approach to conference scheduling that engages the entire community in the planning process. Cobi comprises (a) communitysourcing applications that collect preferences, constraints, and affinity data from community members, and (b) a visual scheduling interface that combines communitysourced data and constraint-solving to enable organizers to make informed improvements to the schedule. This paper describes Cobi's scheduling tool and reports on a live deployment for planning CHI 2013, where organizers considered input from 645 authors and resolved 168 scheduling conflicts. Results show the value of integrating community input with an intelligent user interface to solve complex planning tasks.

Micro-task platforms provide massively parallel, on-demand labor. However, it can be difficult to reliably achieve high-quality work because online workers may behave irresponsibly, misunderstand the task, or lack necessary skills. This paper investigates whether timely, task-specific feedback helps crowd workers learn, persevere, and produce better results. We investigate this question through Shepherd, a feedback system for crowdsourced work. In a between-subjects study with three conditions, crowd workers wrote consumer reviews for six products they own. Participants in the None condition received no immediate feedback, consistent with most current crowdsourcing practices. Participants in the Self-assessment condition judged their own work. Participants in the External assessment condition received expert feedback. Self-assessment alone yielded better overall work than the None condition and helped workers improve over time. External assessment also yielded these benefits. Participants who received external assessment also revised their work more. We conclude by discussing interaction and infrastructure approaches for integrating real-time assessment into online work.

Prototypes ground group communication and facilitate decision making. However, overly investing in a single design idea can lead to fixation and impede the collaborative process. Does sharing multiple designs improve collaboration? In a study, participants created advertisements individually and then met with a partner. In the Share Multiple condition, participants designed and shared three ads. In the Share Best condition, participants designed three ads and selected one to share. In the Share One condition, participants designed and shared one ad. Sharing multiple designs improved outcome, exploration, sharing, and group rapport. These participants integrated more of their partner's ideas into their own subsequent designs, explored a more divergent set of ideas, and provided more productive critiques of their partner's designs. Furthermore, their ads were rated more highly and garnered a higher click-through rate when hosted online.

What authoring possibilities arise by blending machine and human control of live embodied character experiences? This paper explores two different "behind-the-scenes" roles for human operators during a three-month gallery installation of an embodied character experience. In the Transcription role, human operators type players' spoken utterances; then, algorithms interpret the player's intention, choose from pre-authored dialogue based on local and global narrative contexts, and procedurally animate two embodied characters. In the Discourse role, human operators select from semantic categories to interpret player intention; algorithms use this "discourse act" to automate character dialogue and animation. We compare these two methods of blending control using game logs and interviews, and document how the amateur operators initially resisted having to learn the Discourse version, but eventually preferred having the authorial control it afforded. This paper also outlines a design space for blending machine and human control in live character experiences.

Dialogue systems are central to role-playing games, adventure games, interactive fictions, and some forms of interactive drama and cinema---but we have little empirical evidence about how, or even whether, the design of dialogue system interfaces shapes gameplay experience. In this paper we present the results of a study directly comparing three different dialogue system interfaces implemented over the complete course of a dramatic, story-focused game. We find that, holding the rest of the game steady, changing the dialogue interface produces significant changes in gameplay experience. Further, these changes shape perceptions of the system well beyond the interface and its operation. The changes are also sometimes resonant with, and sometimes at odds with, conventional wisdom about these interface options in the game design and writing communities. Our study compared three interfaces: a sentence selection interface (which appears to maximize story involvement), an abstract response menu interface (which maximized reasoning about the underlying game structures), and a natural language understanding interface (which maximized a sense of presence and engagement with the overall experience). Our full results provide valuable future guidance for those seeking a dialogue interface that is resonant with their gameplay experience goals.

Iteration can help people improve ideas. It can also give rise to fixation, continuously refining one option without considering others. Does creating and receiving feedback on multiple prototypes in parallel, as opposed to serially, affect learning, self-efficacy, and design exploration? An experiment manipulated whether independent novice designers created graphic Web advertisements in parallel or in series. Serial participants received descriptive critique directly after each prototype. Parallel participants created multiple prototypes before receiving feedback. As measured by click-through data and expert ratings, ads created in the Parallel condition significantly outperformed those from the Serial condition. Moreover, independent raters found Parallel prototypes to be more diverse. Parallel participants also reported a larger increase in task-specific self-confidence. This article outlines a theoretical foundation for why parallel prototyping produces better design results and discusses the implications for design education.

Iterative prototyping helps designers refine their ideas and discover previously unknown issues and opportunities. However, the time constraints of production schedules can discourage iteration in favor of realization. Is this tradeoff prudent? This paper investigates if - under tight time constraints - iterating multiple times provides more benefit than a single iteration. A between-subjects study manipulates participants' ability to iterate on a design task. Participants in the iteration condition outperformed those in the non-iteration condition. Participants with prior experience with the task performed better. Notably, participants in the iteration condition without prior task experience performed as well as non-iterating participants with prior task experience.

No abstract available.

In this paper we present the results of a qualitative, empirical study exploring the impact of immersive technologies on presence and engagement, using the interactive drama Façade as the object of study. In this drama, players are situated in a married couple's apartment, and interact primarily through conversation with the characters and manipulation of objects in the space. We present participants' experiences across three different versions of Façade -- augmented reality (AR) and two desktop computing based implementations, one where players communicate using speech and the other using typed keyboard input. Through interviews and observations of players, we find that immersive AR can create an increased sense of presence, confirming generally held expectations. However, we demonstrate that increased presence does not necessarily lead to more engagement. Rather, mediation may be necessary for some players to fully engage with certain interactive media experiences.

Our demonstration presents AR Façade, a phyiscally embodied version of the interactive drama Façade, at the Beall Center in Irvine, CA. In this drama, players are situated in a married couple's apartment, and interact primarily through conversation with the characters and manipulation of objects in the space. Our demonstration will include two versions of the experience - an immersive augmented reality (AR) version and a desktop computing based implementation, where players communicate using typed keyboard input. Our recent study cross media study revealled emperical differences between the versions [Dow et al. 2007]. Through interviews and observations of players, we found that immersive AR can create an increased sense of presence, confirming generally held expectations. However, we learned that increased presence does not necessarily lead to more engagement. Rather, mediation may be necessary for some players to fully engage with certain immersive media experiences.

One challenge for ubiquitous computing is providing appropriate tools for professional designers, thus leading to stronger user-valued applications. Unlike many previous tool-builders' attempts to support a specific technology, we take a designer-centered stance, asking the question: how do professional designers externalize ideas for off-the-desktop computing and how do these inform next generation design tools? We report on interviews with designers from various domains, including experience, interaction, industrial, and space designers. The study broadly reveals perceived challenges of moving into a non-traditional design medium, emphasizes the practice of storytelling for relating the context of interaction, and through two case studies, traces the use of various external representations during the design progression of ubicomp applications. Using paperprototyped "walkthroughs" centered on two common design representations (storyboards and physical simulations), we formed a deeper understanding of issues influencing tool development. We offer guidelines for builders of future ubicomp tools, especially early-stage conceptual tools for professional designers to prototype applications across multiple sensors, displays, and physical environments.

We present a concept for augmented reality entertainment, called AR Karaoke, where users perform their favorite dramatic scenes with virtual actors. AR Karaoke is the acting equivalent of traditional Karaoke, where the goal is to facilitate an acting experience for the user that is entertaining for both the user and audience. Prototype implementations were created to evaluate various user interfaces and design approach reveal guidelines that are relevant to the design of mixed reality applications in the domains of gaming, performance, and entertainment.

In this paper [MacIntyre et al 2004]. we describe The Designer's Augmented Reality Toolkit (DART). DART is built on top of Macromedia Director, a widely used multimedia development environment. We summarize the most significant problems faced by designers working with AR in the real world, and discuss how DART addresses them. Most of DART is implemented in an interpreted scripting language, and can be modified by designers to suit their needs. Our work focuses on supporting early design activities, especially a rapid transition from storyboards to working experience, so that the experiential part of a design can be tested early and often. DART allows designers to specify complex relationships between the physical and virtual worlds, and supports 3D animatic actors (informal, sketch-based content) in addition to more polished content. Designers can capture and replay synchronized video and sensor data, allowing them to work off-site and to test specific parts of their experience more effectively.

In this paper we present an approach for exposing tracking technology in an accessible and flexible way to users of a rapid prototyping system for mixed (MR) and augmented reality (AR). Our system provides a tracking framework that alleviates the need for a high level of expertise while also presenting a model of the technology that allows for flexible modification of tracking configurations, the ability to quickly change an application from one type of tracking technology to another, and the creation of synthetic trackers for playback of prerecorded data, data fusion from multiple trackers, and wizard-of-oz applications.

In this paper, we describe The Designer's Augmented Reality Toolkit (DART). DART is built on top of Macromedia Director, a widely used multimedia development environment. We summarize the most significant problems faced by designers working with AR in the real world, and discuss how DART addresses them. Most of DART is implemented in an interpreted scripting language, and can be modified by designers to suit their needs. Our work focuses on supporting early design activities, especially a rapid transition from story-boards to working experience, so that the experiential part of a design can be tested early and often. DART allows designers to specify complex relationships between the physical and virtual worlds, and supports 3D animatic actors (informal, sketch-based content) in addition to more polished content. Designers can capture and replay synchronized video and sensor data, allowing them to work off-site and to test specific parts of their experience more effectively.

This demonstration highlights the Designer's Augmented Reality Toolkit (DART), a system that allows users to easily create augmented reality (AR) experiences. Over the past year our research has focused on the creation of this toolkit that can be used by technologists, designers, and students alike to rapidly prototype AR applications. Current approaches to AR development involve extensive programming and content creation as well as knowledge of technical topics involving cameras, trackers, and 3D geometry. The result is that it is very difficult even for technologists to create AR experiences. Our goal was to eliminate these obstacles that prevent such users from being able to experiment with AR. The DART system is based on the Macromedia Director multimedia-programming environment, the de facto standard for multimedia content creation. DART uses the familiar Director paradigms of a score, sprites and behaviors to allow a user to visually create complex AR applications. DART also provides low-level support for the management of trackers, sensors, and camera via a Director plug-in Xtra. This demonstration will show the wide range of AR and other types of multimedia applications that can be created with DART, and visitors will have the opportunity to use DART to create their own experiences.