Open-ended writing assignments are central to higher education, yet heterogeneous submissions and scale make evaluation difficult. Automated writing evaluation (AWE) promises speed but often trades away transparency and sidelines human judgment. This paper repositions the AI as an on-demand collaborator that can provide specific, targeted support. In a formative study, we expose leverage points in three cognitive dimensions: evidence identification, comparative judgment, and feedback composition. Guided by these insights, we build EvaluAId, which supports interactive rubric-content mapping, adaptive benchmarking and self-calibration, and personalized, rubric-aligned feedback synthesis. Through a within-subjects study with 12 TAs, we evaluate how this approach supports grading compared with a rubric+LLM chatbot and an LLM-based AWE; EvaluAId improved alignment with expert ratings and increased graders’ satisfaction. Finally, interviews with TAs, instructors, and students underscored the value of thoughtfulness supported by EvaluAId while surfacing practical considerations for integration into classroom. Together, our results argue for deliberate, evidence-first, human-in-the-loop evaluation.

Surveys, especially Likert scale questionnaires, are widely used in HCI to capture users’ attitudes and experiences, but numeric ratings alone provide little insight into the rationales behind those ratings. While adding open-ended text fields or post-questionnaire interviews can elicit richer explanations, they often impose extra effort, leading to survey fatigue or recall bias. To address this gap, we proposed Rationalizer, an LLM-supported questionnaire system that generates contextualized rationales to support participants articulate their explanations alongside each Likert item and rating. In a user evaluation comparing it with the traditional questionnaire that included open-ended text fields, Rationalizer increased the percentage of Likert items with rationales, sustained participants’ willingness to provide self-input rationales, and supported them in articulating longer explanations within comparable writing durations as the study progressed. Quality analyses further showed that Rationalizer yielded higher-quality rationales (i.e., justification and relevance) than the traditional questionnaire. These findings highlight the potential of LLM-supported questionnaires to enrich Likert ratings with contextualized, richer explanations.

Advances in AI have opened up the potential for creativity tools to computationally generate design feedback. In a future when designers can request feedback anytime on demand, how would the timing of these requests impact novices’ creative learning processes? What are the tradeoffs of providing access to feedback throughout a design task (in-action) versus only providing feedback after (on-action)? We explored these questions through a Wizard-of-Oz study (N=20) using an interactive design probe, where participants could request feedback either throughout the design process or only after they complete a full draft. We found that in-action participants frequently request feedback, resulting in better improvements as indicated by a greater decrease in issues in their final design. However, we saw that in-action feedback can also risk users overly relying on feedback instead of engaging in more holistic self-evaluation. We discuss the implications of our insights on designing tools for creative feedback.

This three-phase study explores the experiential background of contributors to platforms that provide crowdsourced location-related information. Initially, we utilized interviews to understand users’ expectations for location-related information and the contributors’ experiential background they believe would enhance this information’s utility. We then deployed a survey to identify the top eight sought-after location-information types and their perceived characteristics. Then the concluding online scenario-based study provided quantitative evidence about the interrelationships of eight types of location-related information, ten crucial quality attributes, and aspects of the contributors’ experiential background believed to enhance the utility of the descriptions they provide. Notably, although certain experiential background aspects were deemed universally advantageous across all information types, unique connections were identified among specific information types and distinct experiential background aspects seen as augmenting the contributor’s descriptions’ utility. These insights underline the importance of location-based crowdsourcing platforms incorporating contributors’ experiential background when assigning tasks.

Traditional design galleries enable users to search for examples based on surface attributes (e.g., color or style), and largely obscure underlying principles (e.g., hierarchy or readability). We conducted three studies to explore how galleries could be constructed to help novices learn key design principles. Study 1 revealed that novices gain perspective by observing how designs evolve throughout a process. Study 2 found that novices are better at identifying design issues when viewing iterations that show improvements for just one principle at a time, rather than multiple. Building on these insights, we created ProcessGallery, a tool that enables users to browse contrasting pairs of early-and-late iterations of designs that highlight key improvements organized by design principles. In Study 3, a within-subjects experiment, sixteen participants iterated on a seed design after viewing examples in ProcessGallery versus a traditional gallery. Using ProcessGallery, participants found more appropriate examples, assessed designs better, and preferred ProcessGallery for learning compared to a traditional gallery.

We deployed a feedback visualization tool to learn how students used the tool for interpreting feedback from peers and teaching assistants. The tool visualizes the topic and opinion structure in a collection of feedback and provides interaction for reviewing providers’ backgrounds. A total of 18 teams engaged with the tool to interpret feedback for course projects. We surveyed students (N = 69) to learn about their sensemaking goals, use of the tool to accomplish those goals, and perceptions of specific features. We interviewed students (N = 12) and TAs (N = 2) to assess the tool’s impact on students’ review processes and course instruction. Students discovered valuable feedback, assessed project quality, and justified design decisions to teammates by exploring specific icon patterns in the visualization. The interviews revealed that students mimicked strategies implemented in the tool when reviewing new feedback without the tool. Students found the benefits of the visualization outweighed the cost of labeling feedback.

Examples help students learn insights about key domain principles and processes. However, little is known about how students leverage the Web to discover and learn from examples. In a comparative study, seventy undergraduate students leveraged three types of platforms--- search-based, crit-based, and portfolio-based platforms---to find examples that represent contrasting cases of two design principles. Students reported how each platform's features and mechanisms affected their approach. We identify three main strategies students employed for finding examples on the Web: developing keywords, visually comparing multiple examples, and leveraging community feedback to assess example quality. Our results also indicate that, despite giving access to many examples, none of the existing platforms provide explicit support for learning. We distill three guidelines for creating learner-centered online design galleries to help future learners gain design knowledge.

Causal knowledge is of interest in many areas, such as statistics and machine learning, as it allows people and algorithms to predict outcomes and make data-driven decisions. Researchers in CSCW have proposed tools and workflows to externalize causal knowledge or beliefs from a group of people; however, most of the generated causal diagrams lack a deeper understanding of the causal mechanisms or could not capture diverse beliefs. By integrating narratives with causal diagrams, we implemented an interactive system that allows users to 1) write narratives to rationalize their perceived causal relationships, 2) visualize their causal models using directed diagrams, and 3) review and utilize others' causal diagrams and narratives. We conducted a user study (N=20) to learn how participants leveraged this integrated approach to externalize their perceived causal models for a given application context. Our results showed that the approach implemented in our tool enabled the externalization of users' causal beliefs (e.g., how and why a causal relationship might occur), allowed blind spots of individuals' causal reasoning to be revealed (e.g., learning new ideas from peers), and inspired their causal reasoning (e.g., revising or adding new causal relationships). We also identified the individual differences in people's causal beliefs and observed the impacts of showing others' causal models when one is building his/her causal diagram and narratives. This work provides practical design implications for developing collaborative tools that facilitate capturing and sharing causal beliefs.

Feedback from diverse audiences can vary in focus, differ in structure, and contradict each other, making it hard to interpret and act on. While prior work has explored generating quality feedback, our work helps a designer interpret that feedback. Through a formative study with professional designers (N=10), we discovered that the interpretation process includes categorizing feedback, identifying valuable feedback, and prioritizing which feedback to incorporate in a revision. We also found that designers leverage feedback topic and sentiment, and the status of the provider to aid interpretation. Based on the findings, we created a new tool (Decipher) that enables designers to visualize and navigate a collection of feedback using its topic and sentiment structure. In a preliminary evaluation (N=20), we found that Decipher helped users feel less overwhelmed during feedback interpretation tasks and better attend to critical issues and conflicting opinions compared to using a typical document-editing tool.

Mediation analysis is commonly performed using regressions or Bayesian network analysis in statistics, psychology, and health science; however, it is not effectively supported in existing visualization tools. The lack of assistance poses great risks when people use visualizations to explore causal relationships and make data-driven decisions, as spurious correlations or seemingly conflicting visual patterns might occur. In this paper, we focused on the causal reasoning task over three variables and investigated how an interface could help users reason more efficiently. We developed an interface that facilitates two processes involved in causal reasoning: 1) detecting inconsistent trends, which guides users' attention to important visual evidence, and 2) interpreting visualizations, by providing assisting visual cues and allowing users to compare key visualizations side by side. Our preliminary study showed that the features are potentially beneficial. We discuss design implications and how the features could be generalized for more complex causal analysis.

Online crowd platforms (e.g. social networks, online communities, task markets) enable designers to gain insights from large audiences quickly and affordably. However, there is no guidance for designers to better allocate their social capital, time, and financial resources for acquiring feedback that meets their own needs. Also, feedback received online can be ambiguous and contradictory, making it difficult to interpret and act on. These limitations hinder the utility of crowd feedback, making designers hesitant to actively make use of feedback received. The goal of my dissertation is to 1) formulate a framework that suggests which crowd genres to solicit feedback according to individual needs, 2) develop lightweight activities that promote deeper interpretation on a large volume of feedback, and 3) design and deploy an experimental platform that collects long-term user data, and reduces the burden of conducting online studies of design feedback.

Feedback from diverse audiences can contain ambiguity and contradictions, making it difficult to interpret and act on. To promote deeper interpretation of feedback, we tested the effects of combining a reflection activity and reviewing external feedback for an iterative design task. Designers (N=90) created a design and revised it after a) performing a reflection activity before reviewing feedback, b) performing the reflection after reviewing feedback, c) performing the reflection only, or d) reviewing the feedback only. We measured design quality, depth of revision, perceived effort, and confidence; and categorized the content produced from the reflections. We found that performing reflection after feedback review led to the largest increase in perceived quality for the revised designs, and performing reflection and feedback review regardless of the order resulted in the most extensive revision. Our results also showed that performing the reflection alone yielded outcomes that were similar to when only reviewing feedback, and either activity led to better outcomes than the control condition (no feedback or reflection). Designers stated that the reflection helped them recall their goals, question their choices, and prioritize revisions. We argue that designers should perform a lightweight, explicit reflection to enhance their iterative process, and discuss implications for feedback platforms.

As demand for design education increases, instructors are struggling to provide timely, personalized feedback for student projects. Gathering feedback from classroom peers and external crowds offer scalable approaches, but there is little evidence of how they compare. We report on a study in which students (n=127) created early- and late-stage prototypes as part of nine-week projects. At each stage, students received feedback from peers and external crowds: their own social networks, online communities, and a task market. We measured the quality, quantity and valence of the feedback and the actions taken on it, and categorized its content using a taxonomy of critique discourse. The study found that peers produced feedback that was of higher perceived quality, acted upon more, and longer compared to the crowds. However, crowd feedback was found to be a viable supplement to peer feedback and students preferred it for projects targeting specialized audiences. Feedback from all sources spanned only a subset of the critique categories. Instructors may fill this gap by further scaffolding feedback generation. The study contributes insights for how to best utilize different feedback sources in project-based courses.

Increasingly, designers seek feedback on their designs from crowd platforms such as social networks, Web forums, and paid task markets which demand different amounts of social capital, financial resources, and time. Yet it is unknown how the choice of crowd platform affects feedback generation. We conducted an online study where designers created initial designs and revised the designs based on crowd feedback. We measured the quantity, quality, and content of the feedback received at two iterations and from crowds driven by social status, enjoyment, and financial gain. Our results show, for example, that task markets yield more suggestions, online forums provide more process feedback, and social networks give the most suggestions without payment. We contribute an emergent framework for crowd feedback selection, opportunities for enhancing feedback services, and an experimental platform that researchers can adapt to reduce the burden of conducting online studies of design feedback.