Peer evaluations are a well-established tool for evaluating individual and team performance in collaborative contexts, but are susceptible to social and cognitive biases. Current peer evaluation tools have also yet to address the unique opportunities that online collaborative technologies provide for addressing these biases. In this work, we explore the potential of one such opportunity for peer evaluations: data traces automatically generated by collaborative tools, which we refer to as "activity traces". We conduct a between-subjects experiment with 101 students and MTurk workers, investigating the effects of reviewing activity traces on peer evaluations of team members in an online collaborative task. Our findings show that the usage of activity traces led participants to make more and greater revisions to their evaluations compared to a control condition. These revisions also increased the consistency and participants' perceived accuracy of the evaluations that they received. Our findings demonstrate the value of activity traces as an approach for performing more reliable and objective peer evaluations of teamwork. Based on our findings as well as qualitative analysis of free-form responses in our study, we also identify and discuss key considerations and design recommendations for incorporating activity traces into real-world peer evaluation systems.

The configuration that an instructor enters into an algorithmic team formation tool determines how students are grouped into teams, impacting their learning experiences. One way to decide the configuration is to solicit input from the students. Prior work has investigated the criteria students prefer for team formation, but has not studied how students prioritize the criteria or to what degree students agree with each other. This paper describes a workflow for gathering student preferences for how to weight the criteria entered into a team formation tool, and presents the results of a study in which the workflow was implemented in four semesters of the same project-based design course. In the most recent semester, the workflow was supplemented with an online peer discussion to learn about students' rationale for their selections. Our results show that students want to be grouped with other students who share the same course commitment and compatible schedules the most. Students prioritize demographic attributes next, and then task skills such as programming needed for the project work. We found these outcomes to be consistent in each instance of the course. Instructors can use our results to guide team formation in their own project-based design courses and replicate our workflow to gather student preferences for team formation in any course.

Receiving feedback on preliminary work allows content creators to gain insight and improve outcomes. However, many creators only share in-progress work at late stages of the creative process and lose opportunities to address conceptual issues in the work. To contribute to the base of knowledge of factors that shape one’s decision to seek feedback on their work, we conducted 24 semi-structured interviews with creators in product, interaction and graphic design domains. The results yielded a taxonomy of process-related, social, and cognitive factors that affected a creator’s choice to seek feedback. Next, we administered a survey to quantify the prevalence of these factors at different design stages and for different levels of expertise. Our results show feedback strategy varies by expertise—experts are more likely to create personal deadlines to seek feedback than novices—and by stage. At the early stage, creators sought feedback when testing multiple alternatives. At the late stage, creators are most likely to consider revision granularity prior to seeking feedback. We demonstrate future possibilities for new features in creativity tools through speculative design sketches motivated by our findings.

Receiving feedback on preliminary work allows content creators to gain insight and improve outcomes. However, a lack of commitment for gathering feedback and experiencing evaluation apprehension can delay feedback seeking. In this paper, we operationalize goal setting theory for planning feedback goals and test the effects on feedback seeking and revision. In an online experiment, participants (N=245) wrote an initial story after planning feedback goals (or not), submitted the story for feedback at a time of their choice, and revised the story based on feedback received. Participants anticipated feedback from a supervisor or peer to induce different levels of evaluation apprehension. We found that participants who planned proximal feedback goals sought feedback when their stories were less developed and revised the stories more after receiving feedback compared to when participants planned distant goals. Additionally, participants who anticipated feedback from a supervisor, regardless of goal planning, improved the quality of their stories the most. We did not find that goal setting or the provider's role affected evaluation apprehension. Our findings indicate that content creators should be guided to plan proximal feedback goals to encourage sharing of early drafts of creative work and receive feedback from someone in a position of higher perceived power to foster the most revision and improvement on those drafts.

Feedback is a staple of the design process, but little is known about why designers delay or refuse feedback collection. To fill this knowledge gap, my dissertation identifies triggers and deterrents to feedback seeking behaviors. Based on our findings, I propose and test two interventions to promote feedback seeking behavior: 1) helping designers plan when they seek feedback to increase commitment and 2) generating feedback templates based on design stage to reduce the effort of feedback seeking. We envision a future where creativity support and educational tools use our interventions to encourage designers to seek feedback earlier, more frequently, learn faster, and eventually create better designs.

Advertisers develop algorithms to select the most relevant advertisements for users. However, the opacity of these algorithms, along with their potential for violating user privacy, has decreased user trust and preference in behavioral advertising. To mitigate this, advertisers have started to communicate algorithmic processes in behavioral advertising. However, how revealing parts of the algorithmic process affects users' perceptions towards ads and platforms is still an open question. To investigate this, we exposed 32 users to why an ad is shown to them, what advertising algorithms infer about them, and how advertisers use this information. Users preferred interpretable, non-creepy explanations about why an ad is presented, along with a recognizable link to their identity. We further found that exposing users to their algorithmically-derived attributes led to algorithm disillusionment---users found that advertising algorithms they thought were perfect were far from it. We propose design implications to effectively communicate information about advertising algorithms.

Design instructors are integrating the use of online peer review platforms to keep pace with growing class sizes. However, these platforms typically prioritize randomized peer assignment strategies and show only the current solution to peers when writing feedback. This can result in low quality feedback in project-based design courses. We report on an experiment in which students (N=59) worked on twelve-week design projects and both wrote and received online feedback at four stages. The experiment tested a novel assignment strategy of peer mentorship, where peers were assigned to give feedback to all stages of the same project, and tested showing the context from the preceding design stage when composing feedback. The results showed that displaying the context from the preceding design stage led to feedback with higher perceived quality at the late design stages (but not at earlier stages) and feedback from mentors prompts longer responses from the feedback recipients. Our work contributes deeper empirical understanding of how assignment strategies and showing additional context affects peer feedback and provides practical guidelines for instructors to implement these methods in design courses.