Team roles offer an interpretable lens on collaboration, yet computational studies of roles often rely on domain-specific personas or datadriven clustering rather than theory-grounded taxonomies.We operationalize a taxonomy of eight communication roles grounded in education literature and annotate a corpus of 6,307 Slack messages from 55 students across 18 teams in a semester-long computer science course project.We evaluate whether LLMs can approximate expert labels, enabling scalable, taxonomy-driven role annotation.Using these role labels, we characterize role dynamics over teams' lifecycles, finding that different roles peak at different moments and that students enact a more diverse set of roles as projects progress.To evaluate the utility of our role constructs, we use them to predict peer recognition, outperforming lexical, conversational, and LLM-prompting baselines.To assess generalizability beyond the educational context, we apply the same role constructs to a public dataset (DeliData) to predict team performance improvement after deliberation, again exceeding prior performance.

Teamwork assessment methods, such as peer evaluations, often fail to accurately capture the collaboration processes in team projects. While log data from digital collaboration tools provide objective evidence of contributions, effectively leveraging these data remains challenging. We interviewed 10 instructors and 16 students, then surveyed additional 51 students to identify their valued teamwork behaviors to guide log data usage, and investigate their perceived benefits and concerns. Students prioritized work-related behaviors such as equitable contribution and timeliness, whereas instructors additionally valued interaction behaviors like mutual support. Both groups highlighted significant limitations in raw log data, including omission of offline contributions, insufficient representation of work quality, and misattributing collaborative work to only the person who interacts with the tool. Many students reviewed logs on their own to monitor project progress and workload equity, but lacked structured guidance to meaningfully interpret the insights. Our findings underscore the need for student-centered teamwork assessment approaches that enable students not only to annotate and contextualize their own data, but also to improve their teamwork by reflecting on a rubric of valued teamwork behaviors.

A conversational agent (CA) effectively facilitates online group discussions at scale. However, users may have expectations about how well the CA would perform that do not match with the actual performance, compromising technology acceptance. We built a facilitator CA that detects a member who has low contribution during a synchronous group chat discussion and asks the person to participate more. We designed three techniques to set end-user expectations about how accurately the CA identifies an under-contributing member: 1)information: explicitly communicating the accuracy of the detection algorithm, 2)explanation: providing an overview of the algorithm and the data used for the detection, and 3)adjustment: enabling users to gain a feeling of control over the algorithm. We conducted an online experiment with 163 crowdworkers in which each group completed a collaborative decision-making task and experienced one of the techniques. Through surveys and interviews, we found that the explanation technique was the most effective strategy overall as it reduced user embarrassment, increased the perceived intelligence of the CA, and helped users better understand the detection algorithm. In contrast, the information technique reduced members' contributions and the adjustment technique led to a more negative perceived discussion experience. We also discovered that the interactions with other team members diluted the effects of the techniques on users' performance expectations and acceptance of the CA. We discuss implications for better designing expectation-setting techniques for AI-team collaboration such as ways to improve collaborative decision outcomes and quality of contributions.

Conversational agents (CAs) can analyze online conversations using natural language techniques and effectively facilitate group discussions by sending supervisory messages. However, if a CA makes imperfect interventions, users may stop trusting the CA and discontinue using it. In this study, we demonstrate how inaccurate interventions of a CA and a conversational repair strategy can influence user acceptance of the CA, members' participation in the discussion, perceived discussion experience between the members, and group performance. We built a CA that encourages the participation of members with low contributions in an online chat discussion in which a small group (3-6 members) performs a decision-making task. Two types of errors can occur when detecting under-contributing members: 1) false-positive (FP) errors happen when the CA falsely identifies a member as under-contributing and 2) false-negative (FN) errors occur when the CA misses detecting an under-contributing member. We designed a conversational repair strategy that gives users a chance to contest the detection results and the agent sends a correctional message if an error is detected. Through an online study with 175 participants, we found that participants who received FN error messages reported higher acceptance of the CA and better discussion experience, but participated less compared to those who received FP error messages. The conversational repair strategy moderated the effect of errors such as improving the perceived discussion experience of participants who received FP error messages. Based on our findings, we offer design implications for which model should be selected by practitioners between high precision (i.e., fewer FP errors) and high recall (i.e., fewer FN errors) models depending on the desired effects. When frequent FP errors are expected, we suggest using the conversational repair strategy to improve the perceived discussion experience.

Instructors using algorithmic team formation tools must decide which criteria (e.g., skills, demographics, etc.) to use to group students into teams based on their teamwork goals, and have many possible sources from which to draw these configurations (e.g., the literature, other faculty, their students, etc.). However, tools offer considerable flexibility and selecting ineffective configurations can lead to teams that do not collaborate successfully. Due to such tools' relative novelty, there is currently little knowledge of how instructors choose which of these sources to utilize, how they relate different criteria to their goals for the planned teamwork, or how they determine if their configuration or the generated teams are successful. To close this gap, we conducted a survey (N=77) and interview (N=21) study of instructors using CATME Team-Maker and other criteria-based processes to investigate instructors' goals and decisions when using team formation tools. The results showed that instructors prioritized students learning to work with diverse teammates and performed "sanity checks" on their formation approach's output to ensure that the generated teams would support this goal, especially focusing on criteria like gender and race. However, they sometimes struggled to relate their educational goals to specific settings in the tool. In general, they also did not solicit any input from students when configuring the tool, despite acknowledging that this information might be useful. By opening the "black box" of the algorithm to students, more learner-centered approaches to forming teams could therefore be a promising way to provide more support to instructors configuring algorithmic tools while at the same time supporting student agency and learning about teamwork.

Peer evaluations are a well-established tool for evaluating individual and team performance in collaborative contexts, but are susceptible to social and cognitive biases. Current peer evaluation tools have also yet to address the unique opportunities that online collaborative technologies provide for addressing these biases. In this work, we explore the potential of one such opportunity for peer evaluations: data traces automatically generated by collaborative tools, which we refer to as "activity traces". We conduct a between-subjects experiment with 101 students and MTurk workers, investigating the effects of reviewing activity traces on peer evaluations of team members in an online collaborative task. Our findings show that the usage of activity traces led participants to make more and greater revisions to their evaluations compared to a control condition. These revisions also increased the consistency and participants' perceived accuracy of the evaluations that they received. Our findings demonstrate the value of activity traces as an approach for performing more reliable and objective peer evaluations of teamwork. Based on our findings as well as qualitative analysis of free-form responses in our study, we also identify and discuss key considerations and design recommendations for incorporating activity traces into real-world peer evaluation systems.

We deployed a feedback visualization tool to learn how students used the tool for interpreting feedback from peers and teaching assistants. The tool visualizes the topic and opinion structure in a collection of feedback and provides interaction for reviewing providers’ backgrounds. A total of 18 teams engaged with the tool to interpret feedback for course projects. We surveyed students (N = 69) to learn about their sensemaking goals, use of the tool to accomplish those goals, and perceptions of specific features. We interviewed students (N = 12) and TAs (N = 2) to assess the tool’s impact on students’ review processes and course instruction. Students discovered valuable feedback, assessed project quality, and justified design decisions to teammates by exploring specific icon patterns in the visualization. The interviews revealed that students mimicked strategies implemented in the tool when reviewing new feedback without the tool. Students found the benefits of the visualization outweighed the cost of labeling feedback.

The configuration that an instructor enters into an algorithmic team formation tool determines how students are grouped into teams, impacting their learning experiences. One way to decide the configuration is to solicit input from the students. Prior work has investigated the criteria students prefer for team formation, but has not studied how students prioritize the criteria or to what degree students agree with each other. This paper describes a workflow for gathering student preferences for how to weight the criteria entered into a team formation tool, and presents the results of a study in which the workflow was implemented in four semesters of the same project-based design course. In the most recent semester, the workflow was supplemented with an online peer discussion to learn about students' rationale for their selections. Our results show that students want to be grouped with other students who share the same course commitment and compatible schedules the most. Students prioritize demographic attributes next, and then task skills such as programming needed for the project work. We found these outcomes to be consistent in each instance of the course. Instructors can use our results to guide team formation in their own project-based design courses and replicate our workflow to gather student preferences for team formation in any course.

Feedback such as rubrics, pre-authored statements, and free-form comments are widely deployed in classrooms and feedback exchange communities. Despite these formats having different levels of detail and requiring different composition processes, little research has related these compositional processes for the provider to the feedback perceptions and revision outcomes for the recipient. In an online experiment, we investigated how novice participants (N=285) revised short stories in response to expert feedback composed with four common processes: rubrics, open comments, rubrics with open comments, and rubrics with comments on each rubric item (per-criterion comments). We also surveyed the feedback providers (N=4) to contrast the costs of composing feedback with the benefits of that feedback for the recipients. We found rubrics with per-criterion comments led to the highest revision quality, providers believed rubrics with open comments were most helpful to recipients, and rubrics alone took the least time to compose. Recipients' revision quality and perceptions of feedback helpfulness and credibility increased with feedback detail. We contribute an emergent framework for selecting feedback composition techniques based on attributes of interest and insights linking the composition costs with the benefits of the resulting feedback.

In online group discussions, balanced participation can improve the quality of discussion, members' satisfaction, and positive group dynamics. One approach to achieve balanced participation is to deploy a conversational agent (CA) that encourages participation of under-contributing members, and it is important to design communication strategies of the CA in a way that is supportive to the group. We implemented five communication strategies that a CA can use during a decision-making task in a small group synchronous chat discussion. The five strategies include messages sent to two types of recipients (@username vs. @everyone) crossed by two separate channels (public vs. private), and a peer-mediated strategy where the CA asks a peer to address the under-contributing member. Through an online study with 42 groups, we measured the balance of participation and perceptions about the CA by analyzing chat logs and survey responses. We found that the CA sending messages specifying an individual through a private channel is the most effective and preferred way to increase participation of under-contributing members. Participants also expressed that the peer-mediated strategy is a less intrusive and less embarrassing way of receiving the CA's messages compared to the conventional approach where the CA directly sends a message to the under-contributing member. Based on our findings, we discuss trade-offs of various communication strategies and explain design considerations for building an effective CA that adapts to different group dynamics and situations.

Receiving feedback on preliminary work allows content creators to gain insight and improve outcomes. However, many creators only share in-progress work at late stages of the creative process and lose opportunities to address conceptual issues in the work. To contribute to the base of knowledge of factors that shape one’s decision to seek feedback on their work, we conducted 24 semi-structured interviews with creators in product, interaction and graphic design domains. The results yielded a taxonomy of process-related, social, and cognitive factors that affected a creator’s choice to seek feedback. Next, we administered a survey to quantify the prevalence of these factors at different design stages and for different levels of expertise. Our results show feedback strategy varies by expertise—experts are more likely to create personal deadlines to seek feedback than novices—and by stage. At the early stage, creators sought feedback when testing multiple alternatives. At the late stage, creators are most likely to consider revision granularity prior to seeking feedback. We demonstrate future possibilities for new features in creativity tools through speculative design sketches motivated by our findings.

Sexual harassment (SH) incidents are increasing and call into question the effectiveness of traditional SH prevention training. In this paper, we introduce a proof-of-concept design of a conversational interface (CI) for understanding SH cases. Key features of the interface include that it engages the learner in a dyadic conversation, prompts the learner for guidance, and tells a story of SH from a first-person perspective. From a mixed-methods study (N=32), learners experiencing a SH vignette using the conversational interface reported feeling less overwhelmed with the content, more engaged with the situation, and more comfortable discussing the topic compared to reading the same vignette online. Participants also reported that using a first-person narrative made the vignette feel realistic and relatable. However, there was no difference in empathy between the conditions. We discuss these results and implications for designing effective SH prevention training.

Scaling assessments typically relies on quantifying work quality, yet written comments are the assessment method of choice for open-ended work. Existing scalable solutions compromise by mapping quality scores to pre-authored comments, but how scores influence interpretation of these comments is not well understood. We report results from a study of how 441 participants authored and revised short stories in response to a score, written comments, both types of feedback, or no feedback. We analyzed data from the story-writing task and two surveys to determine task and feedback satisfaction, revision depth and effort, and improvement between drafts for each participant. We found task satisfaction and task performance were positively correlated among participants who were shown a score. Feedback satisfaction, revision effort, and improvement were highest among participants shown written comments. Either type of feedback prompted more deep revisions than no feedback, but together elicited fewer deep revisions than written comments alone. Our work informs the design of scalable open-ended assessment systems by contributing insights regarding how scores influence perceptions of written feedback and subsequent revision outcomes.

Receiving feedback on preliminary work allows content creators to gain insight and improve outcomes. However, a lack of commitment for gathering feedback and experiencing evaluation apprehension can delay feedback seeking. In this paper, we operationalize goal setting theory for planning feedback goals and test the effects on feedback seeking and revision. In an online experiment, participants (N=245) wrote an initial story after planning feedback goals (or not), submitted the story for feedback at a time of their choice, and revised the story based on feedback received. Participants anticipated feedback from a supervisor or peer to induce different levels of evaluation apprehension. We found that participants who planned proximal feedback goals sought feedback when their stories were less developed and revised the stories more after receiving feedback compared to when participants planned distant goals. Additionally, participants who anticipated feedback from a supervisor, regardless of goal planning, improved the quality of their stories the most. We did not find that goal setting or the provider's role affected evaluation apprehension. Our findings indicate that content creators should be guided to plan proximal feedback goals to encourage sharing of early drafts of creative work and receive feedback from someone in a position of higher perceived power to foster the most revision and improvement on those drafts.

Peer evaluations are critical for assessing teams, but are susceptible to bias and other factors that undermine their reliability. At the same time, collaborative tools that teams commonly use to perform their work are increasingly capable of logging activity that can signal useful information about individual contributions and teamwork. To investigate current and potential uses for activity traces in peer evaluation tools, we interviewed (N=11) and surveyed (N=242) students and interviewed (N=10) instructors at a single university. We found that nearly all of the students surveyed considered specific contributions to the team outcomes when evaluating their teammates, but also reported relying on memory and subjective experiences to make the assessment. Instructors desired objective sources of data to address challenges with administering and interpreting peer evaluations, and have already begun incorporating activity traces from collaborative tools into their evaluations of teams. However, both students and instructors expressed concern about using activity traces due to the diverse ecosystem of tools and platforms used by teams and the limited view into the context of the contributions. Based on our findings, we contribute recommendations and a speculative design for a data-centric peer evaluation tool.

Causal knowledge is of interest in many areas, such as statistics and machine learning, as it allows people and algorithms to predict outcomes and make data-driven decisions. Researchers in CSCW have proposed tools and workflows to externalize causal knowledge or beliefs from a group of people; however, most of the generated causal diagrams lack a deeper understanding of the causal mechanisms or could not capture diverse beliefs. By integrating narratives with causal diagrams, we implemented an interactive system that allows users to 1) write narratives to rationalize their perceived causal relationships, 2) visualize their causal models using directed diagrams, and 3) review and utilize others' causal diagrams and narratives. We conducted a user study (N=20) to learn how participants leveraged this integrated approach to externalize their perceived causal models for a given application context. Our results showed that the approach implemented in our tool enabled the externalization of users' causal beliefs (e.g., how and why a causal relationship might occur), allowed blind spots of individuals' causal reasoning to be revealed (e.g., learning new ideas from peers), and inspired their causal reasoning (e.g., revising or adding new causal relationships). We also identified the individual differences in people's causal beliefs and observed the impacts of showing others' causal models when one is building his/her causal diagram and narratives. This work provides practical design implications for developing collaborative tools that facilitate capturing and sharing causal beliefs.

Team formation tools assume instructors should configure the criteria for creating teams, precluding students from participating in a process affecting their learning experience. We propose LIFT, a novel learner-centered workflow where students propose, vote for, and weigh the criteria used as inputs to the team formation algorithm. We conducted an experiment (N=289) comparing LIFT to the usual instructor-led process, and interviewed participants to evaluate their perceptions of LIFT and its outcomes. Learners proposed novel criteria not included in existing algorithmic tools, such as organizational style. They avoided criteria like gender and GPA that instructors frequently select, and preferred those promoting efficient collaboration. LIFT led to team outcomes comparable to those achieved by the instructor-led approach, and teams valued having control of the team formation process. We provide instructors and designers with a workflow and evidence supporting giving learners control of the algorithmic process used for grouping them into teams.

Feedback from diverse audiences can vary in focus, differ in structure, and contradict each other, making it hard to interpret and act on. While prior work has explored generating quality feedback, our work helps a designer interpret that feedback. Through a formative study with professional designers (N=10), we discovered that the interpretation process includes categorizing feedback, identifying valuable feedback, and prioritizing which feedback to incorporate in a revision. We also found that designers leverage feedback topic and sentiment, and the status of the provider to aid interpretation. Based on the findings, we created a new tool (Decipher) that enables designers to visualize and navigate a collection of feedback using its topic and sentiment structure. In a preliminary evaluation (N=20), we found that Decipher helped users feel less overwhelmed during feedback interpretation tasks and better attend to critical issues and conflicting opinions compared to using a typical document-editing tool.

Online feedback exchange platforms enable content creators to collect a diverse set of design feedback quickly. However, creators can experience low quality and harsh feedback when using such platforms. In this paper, we leverage the empathy of the feedback provider to address both these issues. Specifically, we tested two narrative-based empathy arousal interventions: a negative experience and a design process narrative. We also examined whether ingroup framing further enhances the effects of empathy arousal. In a 3x2 online experiment, participants (n=205) wrote feedback on a poster design after experiencing one of the intervention conditions or a control condition. Our results show both the design process narrative and ingroup framing conditions significantly increased the feedback quality and effort invested in the task. The negative experience narrative condition had similar effects and participants reported significantly increased disapproval towards harsh feedback. We discuss the implications of our results for the design of feedback exchange platforms.

In this Research Work in Progress Paper, we examine the effectiveness of CATME, a tool that implements a criteria-based team formation approach. The tool facilitates forming teams based on criteria like demographics, skills, and work styles. This information is collected from the students via an online survey. The effectiveness of this genre of tool depends on the practicality of the instructor's configuration of the criteria, the veracity of students' responses to the survey, and the soundness of the algorithm. In this work-in-progress paper, we investigate potential issues affecting these factors. Our study was conducted by performing new analysis of data collected from a prior study comparing the performance of teams formed using CATME or randomly in a user interface design course. The performance of teams was not statistically different between the two conditions. In examining the students' responses to the team formation survey, we found issues related to Self-Assessment such as inconsistencies between students' ratings of their skills and reporting of their strongest skills. Likewise, we found some cases where the tool produced unexpected results when calculating the homogeneity of the skills of a team. Implications for instructors to mitigate these problems are discussed.

How can instructors group students into teams that interact and learn effectively together? One strand of research advocates for grouping students into teams with "good" compositions such as skill diversity. Another strand argues for deploying team-building activities to foster interpersonal relations like psychological safety. Our work synthesizes these two strands of research. We describe an experiment (N=249) that compares how team composition vs. team-building activities affect student team outcomes. In two university courses, we composed student teams either randomly or using a criteria-based team formation tool. Teams further performed team-building activities that promoted either team or task outcomes. We collected project scores, and used surveys to measure psychological safety, perceived performance, and team satisfaction. Surprisingly, the criteria-based teams did not statistically differ from the random teams on any of the measures taken, despite having compositions that better satisfied the criteria defined by the instructor. Our findings argue that, for instructors deploying a team formation tool, creating an expectation among team members that their team can perform well is as important as tuning the criteria in the tool. We also found that student teams reported high levels of psychological safety, but these levels appeared to develop organically and were not affected by the activities or compositional strategies tested. We distill these and other findings into implications for the design and deployment of team formation tools for learning environments.

Content creators fear receiving unnecessarily harsh criticism when posting creative work in online platforms. We refer to feedback written in an unnecessary harsh tone as negative feedback. We conducted an online experiment to investigate the efficacy of three coping activities for mitigating the influence of negative feedback: self-affirmation, expressive writing, and distraction. Participants (N=480) received feedback sets with different balances of neutral and negative valence content and revised their essays after performing the assigned activity. We measured participants' affective states, extents of revision, and their perceptions of the feedback and its providers. Our results showed even a small amount of negativity had significant adverse effects on all the measures. For the coping activities, we found that expressive writing encouraged essay revision, distraction improved affective states and feedback provider perception, and self-affirmation had no significant effects on the measures. Our results contribute further empirical knowledge of how negative valence feedback impacts content creators and how the coping activities tested mitigate these effects. We also offer practical guidelines regarding when and how to use the activities tested in online feedback platforms.

Iteration prompted by feedback is fundamental to creative design. Designers often use online communities to gather feedback, but there is little knowledge of how designers use these communities to iterate on their work. We studied iterative design practice in three active forums that target novice design critique in a popular online community. We collected a large corpus of projects (n=3,730) and comments (n=29,412). The data was analyzed to determine how often designers iterate on their projects, and which characteristics of the feedback correlate with iteration. Two surveys (n=38) were conducted to further learn how often and at which design stages designers post their work to the community. Our results showed that more and longer comments, comments with less positive tone and comments with more thought-provoking statements correlated with iteration. Our analysis also showed that two or more iterations were shared for only 21% of the projects. The survey results explained that the rate of iteration was due in part to designers sharing their projects with the community near the end of their design process. Our findings also suggest how online communities can better support iterative design practice.

Content creators are less receptive to feedback with negative valence, and such feedback is frequently received online. To address this problem, we propose a novel method that orders a set of feedback based on its valence; using the feedback with positive valence to mitigate the effects of the negative valence feedback. To test the method, participants (N=270) wrote a story for children based on a given illustration and then revised their story after receiving a set of feedback. The feedback set was delivered with different valence orders and with different source identity cues. We measured participants' affective states, perceptions of the feedback and its source, revision extents, and story quality. Our main result is that presenting negative feedback last improved content creators' affective states and their perception of the feedback set relative to placing the negative feedback in other positions. This pattern was consistent across all feedback source conditions. The work contributes a simple and novel way to order a set of feedback that improves feedback receptivity.

Feedback from diverse audiences can contain ambiguity and contradictions, making it difficult to interpret and act on. To promote deeper interpretation of feedback, we tested the effects of combining a reflection activity and reviewing external feedback for an iterative design task. Designers (N=90) created a design and revised it after a) performing a reflection activity before reviewing feedback, b) performing the reflection after reviewing feedback, c) performing the reflection only, or d) reviewing the feedback only. We measured design quality, depth of revision, perceived effort, and confidence; and categorized the content produced from the reflections. We found that performing reflection after feedback review led to the largest increase in perceived quality for the revised designs, and performing reflection and feedback review regardless of the order resulted in the most extensive revision. Our results also showed that performing the reflection alone yielded outcomes that were similar to when only reviewing feedback, and either activity led to better outcomes than the control condition (no feedback or reflection). Designers stated that the reflection helped them recall their goals, question their choices, and prioritize revisions. We argue that designers should perform a lightweight, explicit reflection to enhance their iterative process, and discuss implications for feedback platforms.

Many students underperform on exams due to experiencing high test anxiety. We report on a study comparing a novel intervention of seeking support from one's social network to the more common approaches of expressive writing and studying task-relevant materials for simulated open-ended test questions. We measured in-the-moment (state) anxiety before and after each intervention, and correctness of the solutions. We also surveyed students to learn about their perceptions of the interventions. Our results showed that social support decreased the anxiety of high test-anxious students by 21% with the reduction in anxiety correlating with the number of messages received. Social support also allowed high test-anxious students to score at the level of low test-anxious students. Expressive writing showed a similar effect, but increased the anxiety of low test-anxious students by 61%. Studying task materials had no effect on anxiety and high test-anxious students performed worse than low test-anxious students. Despite benefiting from social support, we found that students were uncomfortable soliciting support from their online social network. Realizing the benefits of this approach may therefore require different formulations of social support in practice.

To meet the demand for authentic, timely, and affordable feedback, researchers have explored technologies to connect designers with feedback providers online. While researchers have implemented mechanisms to improve the content of feedback, most systems for online feedback exchange do not support an end-to-end cycle, from help-seeking to sense-making to action. Building on extant literature in learning sciences, design, organizational behavior, and online communities, we propose a conceptual framework to highlight critical processes that affect online feedback exchange. We contribute research questions for future feedback systems and argue that online feedback systems must be able to support designers through five activities that happen before, during, and after the feedback exchange. Our framework suggests that systems should address broader socio-psychological factors, such as how intent should be communicated online, how dialogue can support the interpretation of feedback, and how to balance the tradeoffs of anonymizing feedback providers.

Instructors are increasingly using algorithmic tools for team formation, yet little is known about how these tools are applied or how students and instructors perceive their use. We studied a representative team formation tool (CATME) in eight project-based courses. An instructor uses the tool to form teams by surveying students' working styles, skills, and demographics; then configuring these criteria as input into an algorithm that assigns teams. We surveyed students (N=277) in the courses to gauge their perceptions of the strengths and weaknesses of the tool and ideas for improving it. We also interviewed instructors (N=13) different from those who taught the eight courses to learn about their criteria selections and perceptions of the tool. Students valued the rational basis for forming teams but desired a stronger voice in criteria selection and explanations as to why they were assigned to a particular team. Instructors appreciated the efficiency of team formation but wanted to view exemplars of criteria used in similar courses. This work contributes recommendations for deploying team formation tools in educational settings and for better satisfying the goals of all stakeholders.

Designers are increasingly leveraging online crowds; yet, online contributors may lack the expertise, context, and sensitivity to provide effective critique. Rubrics help feedback providers but require domain experts to write them and may not generalize across design domains. This paper introduces and tests a novel semi-automated method to support feedback providers by analyzing feedback language. In our first study, 52 students from two design courses created design solutions and received feedback from 176 online providers. Instructors, students, and crowd contributors rated the helpfulness of each feedback response. From this data, an algorithm extracted a set of natural language features (e.g., specificity, sentiment etc.) that correlated with the ratings. The features accurately predicted the ratings and remained stable across different raters and design solutions. Based on these features, we produced a critique style guide with feedback examples - automatically selected for each feature - to help providers revise their feedback through self-assessment. In a second study, we tested the validity of the guide through a between-subjects experiment (n=50). Providers wrote feedback on design solutions with or without the guide. Providers generated feedback with higher perceived helpfulness when using our style-based guidance.

As demand for design education increases, instructors are struggling to provide timely, personalized feedback for student projects. Gathering feedback from classroom peers and external crowds offer scalable approaches, but there is little evidence of how they compare. We report on a study in which students (n=127) created early- and late-stage prototypes as part of nine-week projects. At each stage, students received feedback from peers and external crowds: their own social networks, online communities, and a task market. We measured the quality, quantity and valence of the feedback and the actions taken on it, and categorized its content using a taxonomy of critique discourse. The study found that peers produced feedback that was of higher perceived quality, acted upon more, and longer compared to the crowds. However, crowd feedback was found to be a viable supplement to peer feedback and students preferred it for projects targeting specialized audiences. Feedback from all sources spanned only a subset of the critique categories. Instructors may fill this gap by further scaffolding feedback generation. The study contributes insights for how to best utilize different feedback sources in project-based courses.

Design instructors are integrating the use of online peer review platforms to keep pace with growing class sizes. However, these platforms typically prioritize randomized peer assignment strategies and show only the current solution to peers when writing feedback. This can result in low quality feedback in project-based design courses. We report on an experiment in which students (N=59) worked on twelve-week design projects and both wrote and received online feedback at four stages. The experiment tested a novel assignment strategy of peer mentorship, where peers were assigned to give feedback to all stages of the same project, and tested showing the context from the preceding design stage when composing feedback. The results showed that displaying the context from the preceding design stage led to feedback with higher perceived quality at the late design stages (but not at earlier stages) and feedback from mentors prompts longer responses from the feedback recipients. Our work contributes deeper empirical understanding of how assignment strategies and showing additional context affects peer feedback and provides practical guidelines for instructors to implement these methods in design courses.

Increasingly, designers seek feedback on their designs from crowd platforms such as social networks, Web forums, and paid task markets which demand different amounts of social capital, financial resources, and time. Yet it is unknown how the choice of crowd platform affects feedback generation. We conducted an online study where designers created initial designs and revised the designs based on crowd feedback. We measured the quantity, quality, and content of the feedback received at two iterations and from crowds driven by social status, enjoyment, and financial gain. Our results show, for example, that task markets yield more suggestions, online forums provide more process feedback, and social networks give the most suggestions without payment. We contribute an emergent framework for crowd feedback selection, opportunities for enhancing feedback services, and an experimental platform that researchers can adapt to reduce the burden of conducting online studies of design feedback.

Crowd feedback services offer a new method for acquiring feedback during design. A key problem is that the services only return the feedback without any cues about the people who provided it. In this paper, we investigate two cues of a feedback provider -- the effort invested in a feedback task and expertise in the domain. First, we tested how positive and negative cues of a provider's effort and expertise affected perceived quality of the feedback. Results showed both cues affected perceived quality, but primarily when the cues were negative. The results also showed that effort cues affected perceived quality as much as expertise. In a second study, we explored the use of behavioral data for modeling effort for feedback tasks. For a binary classification, the models achieved up to 92% accuracy relative to human raters. This result validates the feasibility of implementing effort cues in crowd services. The contributions of this work will enable increased transparency in crowd feedback services, benefiting both designers and feedback providers.

The modality students use to complete programming-centric exams is shifting from paper to computer. We developed and distributed a survey to investigate how exam modality affects student perceptions of test anxiety and performance during programming-centric exams. We examine the distribution of modality preference among students and understand how students cope with test anxiety. We found a majority of students report at least moderate perceived anxiety (rating -- 4 on 7-pt scale) on paper- (69%) and computer-based (64%) exams. Though moderate anxiety was experienced by a majority of the students, we found 1 in 5 students had no strategy to cope with anxiety. Additionally, we examine how students perceived anxiety during testing affects their thought process and behavior. We found computer-based exams were perceived as more beneficial to the quality and speed of student's solutions and a majority of students preferred computer-based testing (67%). Lastly, we introduce possible technological solutions to benefit students experiencing test anxiety while not impeding less anxious students.

Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a web-based system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer's experience and benefits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services.

Consensus is a desired but elusive goal in many distributed discussions. A critical problem is that discussion platforms lack mechanisms for realizing consensus strategies and realizing these strategies without tool support can be hard. This paper introduces Procid, a novel browser plugin that provides interaction and visualization features for bringing consensus strategies to distributed design discussions. Key features include the ability to organize discussions around ideas, to register and visualize support for or against ideas, and to define criteria for evaluating ideas. It also applies interaction constraints fostering best practices of consensus building. Procid extends the discussion platform of one open source software community. Two evaluations were conducted. The first collected perceptions of the tool from members of the community for their own discussions. The second compared how Procid affects a distributed design discussion relative to the current discussion platform in the community. Results of both studies showed that users found the features of our tool beneficial and perceived it as more effective for consensus building than the existing platform.

Crowd feedback systems offer designers an emerging approach for improving their designs, but there is little empirical evidence of the benefit of these systems. This paper reports the results of a study of using a crowd feedback system to iterate on visual designs. Users in an introductory visual design course created initial designs satisfying a design brief and received crowd feedback on the designs. Users revised the designs and the system was used to generate feedback again. This format enabled us to detect the changes between the initial and revised designs and how the feedback related to those changes. Further, we analyzed the value of crowd feedback by comparing it with expert evaluation and feedback generated via free-form prompts. Results showed that the crowd feedback system prompted deep and cosmetic changes and led to improved designs, the crowd recognized the design improvements, and structured workflows generated more interpretative, diverse and critical feedback than free-form prompts.

Despite software history being indispensable for developers, there is little empirical knowledge about how they examine software history. Without such knowledge, researchers and tool builders are in danger of making wrong assumptions and building inadequate tools. In this paper we present an in-depth empirical study about the motivations developers have for examining software history, the strategies they use, and the challenges they encounter. To learn these, we interviewed 14 experienced developers from industry, and then extended our findings by surveying 217 developers. We found that history does not begin with the latest commit but with uncommitted changes. Moreover, we found that developers had different motivations for examining recent and old history. Based on these findings we propose 3-LENS HISTORY, a novel unified model for reasoning about software history.

Typically video projectors display images onto white screens, which can result in a washed out image. Projectibles algorithmically control the display surface color to increase the contrast and resolution. By combining a printed image with projected light, we can create animated, high resolution, high dynamic range visual experiences for video sequences. We present two algorithms for separating an input video sequence into a printed component and projected component, maximizing the combined contrast and resolution while minimizing any visual artifacts introduced from the decomposition. We present empirical measurements of real-world results of six example video sequences, subjective viewer feedback ratings, and we discuss the benefits and limitations of Projectibles. This is the first approach to combine a static display with a dynamic display for the display of video, and the first to optimize surface color for projection of video.

Hundreds of thousands of crowdfunding campaigns have been launched, but more than half of them have failed. To better understand the factors affecting campaign outcomes, this paper targets the content and usage patterns of project updates -- communications intended to keep potential funders aware of a campaign's progress. We analyzed the content and usage patterns of a large corpus of project updates on Kickstarter, one of the largest crowdfunding platforms. Using semantic analysis techniques, we derived a taxonomy of the types of project updates created during campaigns, and found discrepancies between the design intent of a project update and the various uses in practice (e.g. social promotion). The analysis also showed that specific uses of updates had stronger associations with campaign success than the project's description. Design implications were formulated from the results to help designers better support various uses of updates in crowdfunding campaigns.

Feedback on designs is critical for helping users iterate toward effective solutions. This paper presents Voyant, a novel system giving users access to a non-expert crowd to receive perception-oriented feedback on their designs from a selected audience. Based on a formative study, the system generates the elements seen in a design, the order in which elements are noticed, impressions formed when the design is first viewed, and interpretation of the design relative to guidelines in the domain and the user's stated goals. An evaluation of the system was conducted with users and their designs. Users reported the feedback about impressions and interpretation of their goals was most helpful, though the other feedback types were also valued. Users found the coordinated views in Voyant useful for analyzing relations between the crowd's perception of a design and the visual elements within it. The cost of generating the feedback was considered a reasonable tradeoff for not having to organize critiques or interrupt peers.

Expanding and improving design knowledge is a vital part of higher education due to the growing demand for employees who can think both critically and creatively. However, developing effective methods for assessing what students have learned in design courses is one of the most elusive challenges of design education due to the subjective nature of design. For example, evaluating design outcomes is problematic due to the common pattern of increasing enrollments and reduced resources for design instruction. In this article, we propose and evaluate a new assessment method that uses a novel application of Bayesian Truth Serum (BTS), a scoring algorithm, in order to provide a scalable and reliable measure of design knowledge. This method requires no subjective input from the design instructor, nor does it require answers to questions that have distinct right or wrong answers. We tested this method over a 4-week period with 71 design students in an upper-level design course. For the study, participants were asked to provide responses to multiple-choice BTS survey questions, generate ideas for a design problem, and provide feedback on other participants' ideas. The survey data were used to calculate BTS indices of expertise and statistical tests were performed to determine how the indices correlated with participant ideation and critique proficiency. The results from this study show a modest correlation between the BTS indices of expertise and later performance on generative design tasks and a correlation between the students' ability to critique designs and their BTS scores. These findings suggest that the BTS assessment method can be used to supplement existing evaluation practices for individual design assessment, particularly in courses where group projects are used as the primary means of evaluation. In addition, the results show promise for using the BTS method in classes where design projects or design critiques are not feasible due to time constraints or large class sizes.

Reflection-in- and -on-action are key elements of creative design. While reflection-in-action has been often studied, reflection-on-action has received less attention. In this paper, we fill this gap by first reporting results from in-depth interviews (N=12) with practicing designers aimed at understanding activities related to reflection-on-action. We found that activities related to reflection-on-action are intentional, repetitive and frequent, design materials are the primary aids for reflective activities, and there is a strong need for better reflection support tools. To address this need, we designed, implemented and evaluated a novel interactive visualization tool -- ReflectionSpace. The tool uses file meta-data and naming conventions to map design materials to the appropriate design phase and context of use and places corresponding representations in a time- and activity-centric visualization that can be navigated at different levels of detail. User feedback from a comparative study reveals that such a representation of design process is more beneficial and preferred relative to the traditional file-centric approach for fostering reflection-on-action.

We present BeThere, a proof-of-concept system designed to explore 3D input for mobile collaborative interactions. With BeThere, we explore 3D gestures and spatial input which allow remote users to perform a variety of virtual interactions in a local user's physical environment. Our system is completely self-contained and uses depth sensors to track the location of a user's fingers as well as to capture the 3D shape of objects in front of the sensor. We illustrate the unique capabilities of our system through a series of interactions that allow users to control and manipulate 3D virtual content. We also provide qualitative feedback from a preliminary user study which confirmed that users can complete a shared collaborative task using our system.

CrowdBand, a sound composition system, demonstrates how a crowd can create works that meet requested criteria and fulfill the aesthetic character given by keyword description and examples. CrowdBand allows flexibility in music composition in terms of duration of the music, completion time and cost of music composition by giving the requestor two modes - thrifty and normal. CrowdBand’s workflow divides the composition task into three sections: requesting fundamental sounds, assembling sounds into compositions, and evaluating the results. Based on the crowd workers’ responses, we conclude that crowdsourced workers who are non-musicians can design sound and create novel sound compositions through CrowdBand. We also conclude that CrowdBand gives the musically-untrained crowd workers the ability to use common compositional techniques, such as sound layering, vertical stacking of sounds to create harmonic effects, related melodic lines (contrapuntal techniques), and transitions between aesthetic notions, or sound themes. Finally, we show improved, faster results with successive simplification and examples.

We report results of a study which examines consensus building in user interface design discussions in open source software communities. Our methodology consisted of conducting interviews with designers and developers from the Drupal and Ubuntu communities (N=17) and analyzing a large corpus of interaction data collected from Drupal. The interviews captured user perspectives on the challenges of reaching consensus, techniques employed for building consensus, and the consequences of not reaching consensus. We analyzed the interaction data to determine how different elements of the content, process, and user relationships in the design discussions affect consensus. Our main result shows that design discussions engaging participants with more experience and prior interaction history are more likely to reach consensus. Based on all of our results, we formulated design implications for promoting consensus in distributed discussions of user interface design issues.

Critique is an indispensible part of creative work and many online communities have formed for this shared purpose. As design choices within the communities can impact the effectiveness of the critiques produced, it is important to study these communities and offer guidance for decisions. In this paper, we report the results of a case study exploring one large online community dedicated to critique in the domain of digital photography. We analyzed a large corpus of interaction data to understand the benefit of participation, the response dynamics, factors predicting critique ratings, and patterns of reciprocal interaction. Interviews with users were also conducted to uncover motives for participation and expectations of the critiques within the community. The results and insights gained from this work were distilled into recommendations for improving the design of systems that support community-based critique of creative artifacts.

Idea pipelines enable open innovation within organizations but require the evaluation teams to assess large numbers of ideas. To help filter promising ideas, community voting is often included as part of the pipeline but the outcome of the voting rarely aligns with the ideas selected by the team. To address this problem, we introduce a new scoring model for increasing the findability of promising ideas within idea pipelines. In the model, each participant need only score a subset of the ideas, ideas are scored independently, and the individual scores can be aggregated. We tested the model on an authentic data set and found our model filters ideas chosen by an evaluation team better than community votes.

Though refactoring tools have been available for more than a decade, research has shown that programmers underutilize such tools. However, little is known about why programmers do not take advantage of these tools. We have conducted a field study on programmers in their natural settings working on their code. As a result, we collected a set of interaction data from about 1268 hours of programming using our minimally intrusive data collectors. Our quantitative data show that programmers prefer lightweight methods of invoking refactorings, usually perform small changes using the refactoring tool, proceed with an automated refactoring even when it may change the behavior of the program, and rarely preview the automated refactorings. We also interviewed nine of our participants to provide deeper insight about the patterns that we observed in the behavioral data. We found that programmers use predictable automated refactorings even if they have rare bugs or change the behavior of the program. This paper reports some of the factors that affect the use of automated refactorings such as invocation method, awareness, naming, trust, and predictability and the major mismatches between programmers' expectations and automated refactorings. The results of this work contribute to producing more effective tools for refactoring complex software.

In this paper we study the design space of free-space interactions for multiscale navigation afforded by mobile depth sensors. Such interactions will have a greater working volume, more fluid control and avoid screen occlusion effects intrinsic to touch screens. This work contributes the first study to show that mobile free-space interactions can be as good as touch. We also analyze sensor orientation and interaction volume usage, resulting in strong implications for how sensors should be placed on mobile devices. We describe a user study evaluating mobile free-space navigation techniques and the impacts of sensor orientation on user experience. Finally, we discuss guidelines for future mobile free-space interaction techniques and sensor design.

Reflection is an integral part of the creative design process. However, reflection is often considered as an enigmatic, un-orderly, and irregular process, leading to a lack of support from existing design and creativity support tools. Through a contextual inquiry (N=12) of designers' reflection practice, we found that reflection is predominantly a systematic, intentional, and repeatedly practiced activity. We also found that artifacts play an imperative role in supporting reflective process. We offer guidelines for the design of better reflection support tools.

Establishing a grassroots innovation pipeline has come to the fore as strategy for nurturing innovation within large organizations. A key element of such pipelines is the use of an idea management system that enables and encourages community ideation on defined business problems. The value of these systems can be highly sensitive to design choices, as different designs may influence participation. We report the results of a case study examining the use of one particular idea management system and pipeline. We analyzed the content, interaction, and participation from three creativity challenges organized via the pipeline and conducted interviews with users to uncover motivations for participating and perceptions of the outcomes. Additional interviews were conducted with senior managers to learn about the objectives, successes, and unique nature of the pipeline. From the results, we formulate recommendations for improving the design of idea management systems and execution of the pipelines within organizations.

We explore interacting with everyday objects by representing content as interactive surface particles. Users can build their own physical world, map virtual content onto their physical construction and play directly with the surface using a stylus. A surface particle representation allows programmed content to be created independent of the display object and to be reused on many surfaces. We demonstrated this idea through a projector-camera system that acquires the object geometry and enables direct interaction through an IR tracked stylus. We present three motivating example applications, each displayed on three example surfaces. We discuss a set of interaction techniques that show possible avenues for structuring interaction on complicated everyday objects, such as Surface Adaptive GUIs for menu selection. Through a preliminary informal evaluation and interviews with end users, we demonstrate the potential of interacting with surface particles and identify improvements necessary to make this interaction practical on everyday surfaces.

A notification represents the proactive delivery of information to a user and reduces the need to visually scan or repeatedly check an external information source. At the same time, notifications often interrupt user tasks at inopportune moments, decreasing productivity and increasing frustration. Controlled studies have shown that linking notification delivery to the perceptual structure of a user's tasks can reduce these interruption costs. However, in these studies, the scheduling was always performed manually, and it was not clear whether it would be possible for a system to mimic similar techniques. This article contributes the design and implementation of a novel system called Oasis that aligns notification scheduling with the perceptual structure of user tasks. We describe the architecture of the system, how it detects task structure on the fly without explicit knowledge of the task itself, and how it layers flexible notification scheduling policies on top of this detection mechanism. The system also includes an offline tool for creating customized statistical models for detecting task structure. The value of our system is that it intelligently schedules notifications, enabling the reductions in interruption costs shown within prior controlled studies to now be realized by users in everyday desktop computing tasks. It also provides a test bed for experimenting with how notification management policies and other system functionalities can be linked to task structure.

Generating and managing multiple ideas is a fundamental part of the creative process for both individual and teams of designers. To be useful for early design, computer tools must effectively support this common practice. This paper proposes a demonstration of a new collaborative interface for managing ideas in computer-based design tools. The core of the interface provides interactive spatial maps for creating, organizing, and reflecting on ideas. The interface was also substantially revised in response to lessons learned from a study comparing its use to the use of other tools for managing multiple ideas during early design work. The revised interface allows designers to tag and filter the idea space, arrange ideas in three distinct views, and efficiently compose and decompose content from multiple ideas.

The use of examples serves a critical role in creative design practice, but details of this process remain an enigma. This is problematic for both the understanding of design activity as well as for developing more effective design tools. In this paper, we report results of a study that understands and compares how designers (N=11) utilize, manage, and share examples to support the creative design process. The domains studied were Web, graphic, and product design. Our study shows that examples are a cornerstone of creative practice and are utilized for many reasons throughout the design process. Since examples are pivotal to the success of a project, more effective tools that support retrieval, storage, and dissemination of examples are needed. This paper contributes understanding of the benefits and roles of examples in the design process and implications for the design of more effective tools that support example usage.

Prior knowledge is a critical resource for design, especially when designers are striving to generate new ideas for complex problems. Systems that improve access to relevant prior knowledge and promote reuse can improve design efficiency and outcomes. Unfortunately, such systems have not been widely adopted indicating that user needs in this area have not been adequately understood. In this paper, we report the results of a contextual inquiry into the practices of and attitudes toward knowledge management and reuse during early design. The study consisted of interviews and surveys with professional designers in the creative domains. A novel aspect of our work is the focus on early design, which differs from but complements prior works' focus on knowledge reuse during later design and implementation phases. Our study yielded new findings and implications that, if applied, will help bring the benefits of knowledge management systems and reuse into early design activity.

We present a new interaction framework for collaborating in multiple display environments (MDEs) and report results from a field study investigating its use in an authentic work setting. Our interaction framework, IMPROMPTU, allows users to share task information across displays via off-the-shelf applications, to jointly interact with information for focused problem solving and to place information on shared displays for discussion and reflection. Our framework also includes a lightweight interface for performing these and related actions. A three week field study of our framework was conducted in the domain of face-to-face group software development. Results show that teams utilized almost every feature of the framework in support of a wide range of development-related activities. The framework was used most to facilitate opportunistic collaboration involving task information. Teams reported wanting to continue using the framework as they found value in it overall.

We present a novel system for notification management and report results from two studies testing its performance and impact. The system uses statistical models to realize defer-to-breakpoint policies for managing notifications. The first study tested how well the models detect three types of breakpoints within novel task sequences. Results show that the models detect breakpoints reasonably well, but struggle to differentiate their type. Our second study explored effects of managing notifications with our system on users and their tasks. Results showed that scheduling notifications at breakpoints reduces frustration and reaction time relative to delivering them immediately. We also found that the relevance of notification content determines the type of breakpoint at which it should be delivered. The core concept of scheduling notifications at breakpoints fits well with how users prefer notifications to be managed. This indicates that users would likely adopt the use of notification management systems in practice.

Notifications can have reduced interruption cost if delivered at moments of lower mental workload during task execution. Cognitive theorists have speculated that these moments occur at subtask boundaries. In this article, we empirically test this speculation by examining how workload changes during execution of goal-directed tasks, focusing on regions between adjacent chunks within the tasks, that is, the subtask boundaries. In a controlled experiment, users performed several interactive tasks while their pupil dilation, a reliable measure of workload, was continuously measured using an eye tracking system. The workload data was extracted from the pupil data, precisely aligned to the corresponding task models, and analyzed. Our principal findings include (i) workload changes throughout the execution of goal-directed tasks; (ii) workload exhibits transient decreases at subtask boundaries relative to the preceding subtasks; (iii) the amount of decrease tends to be greater at boundaries corresponding to the completion of larger chunks of the task; and (iv) different types of subtasks induce different amounts of workload. We situate these findings within resource theories of attention and discuss important implications for interruption management systems.

research-article Share on Special section from the ACM multimedia conference 2007 Authors: Brian P. Bailey University of Illinois University of IllinoisView Profile , Nicu Sebe University of Amsterdam University of AmsterdamView Profile , Alan Hanjalic Delft University of Technology Delft University of TechnologyView Profile Authors Info & Claims ACM Transactions on Multimedia Computing, Communications, and ApplicationsVolume 5Issue 1October 2008 Article No.: 1pp 1–3https://doi.org/10.1145/1404880.1404881Published:30 October 2008Publication History 0citation198DownloadsMetricsTotal Citations0Total Downloads198Last 12 Months4Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my Alerts New Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Creativity support tools are set an especially difficult task when they are applied to art/science collaboration. Not because of any fundamental incompatibility between the disciplines, but because creativity support tools are rarely supple enough to manage dramatically shifting requirements at various stages of design or handle the diversity of artifacts that might be generated. Traditional methods of evaluation of collaborative support tools may not address these aspects. This workshop aims to examine three specific areas open to expanded modes of evaluation; the social aspects of tools and tool use, how artifacts are created and manipulated in support tools, and how the expanding contexts of art/science collaborations may be rapidly changing support tool requirements.

Informal design tools can provide immense value during the creative stages of the design process, e.g., by transforming sketches into interactive simulations. Two key limitations of informal and many other design tools are that they do not promote working with multiple design ideas in parallel or collaboration. In this paper, we present a new interaction model that allows a team of designers to work efficiently with multiple ideas in parallel. The model is grounded in theories of creativity and collaboration, and was further informed by observations of creative group work practice. Our interaction model is fully demonstrated within a new system called TEAM STORM. Results from an initial evaluation indicate that design teams are able to effectively utilize our system to create, organize, and share multiple design ideas during creative group work. The benefit of our model is that it demonstrates how many existing single-user design tools can be extended to support working efficiently with multiple ideas in parallel and co-located collaboration.

The ability to detect and differentiate breakpoints during task execution is critical for enabling defer-to-breakpoint policies within interruption management. In this work, we examine the feasibility of building statistical models that can detect and differentiate three granularities (types) of perceptually meaningful breakpoints during task execution, without having to recognize the underlying tasks. We collected ecological samples of task execution data, and asked observers to review the interaction in the collected videos and identify any perceived breakpoints and their type. Statistical methods were applied to learn models that map features of the interaction to each type of breakpoint. Results showed that the models were able to detect and differentiate breakpoints with reasonably high accuracy across tasks. Among many uses, our resulting models can enable interruption management systems to better realize defer-to-breakpoint policies for interactive, free-form tasks.

Intrusion detection (ID) is one of network security engineers' most important tasks. Textual (command-line) and visual interfaces are two common modalities used to support engineers in ID. We conducted a controlled experiment comparing a representative textual and visual interface for ID to develop a deeper understanding about the relative strengths and weaknesses of each. We found that the textual interface allows users to better control the analysis of details of the data through the use of rich, powerful, and flexible commands while the visual interface allows better discovery of new attacks by offering an overview of the current state of the network. With this understanding, we recommend designing a hybrid interface that combines the strengths of textual and visual interfaces for the next generation of tools used for intrusion detection.

Requests for communication via mobile devices can be disruptive to the current task or social situation. To reduce the frequency of disruptive requests, one promising approach is to provide callers with cues of a receiver's context through an awareness display, allowing informed decisions of when to call. Existing displays typically provide cues based on what can be readily sensed, which may not match what is needed during the call decision process. In this paper, we report results of a four week diary study of mobile phone usage, where users recorded what context information they considered when making a call, and what information they wished others had considered when receiving a call. Our results were distilled into lessons that can be used to improve the design of awareness displays for mobile devices, e.g., show frequency of a receiver's recent communication and distance from a receiver to her phone. We discuss technologies that can enable cues indicated in these lessons to be realized within awareness displays, as well as discuss limitations of such displays and issues of privacy.

Pupillary response is a valid indicator of mental workload and is being increasingly leveraged to identify lower cost moments for interruption, evaluate complex interfaces, and develop further understanding of psychological processes. Existing tools are not sufficient for analyzing this type of data, as it typically needs to be analyzed in relation to the corresponding task’s execution. To address this emerging need, we have developed a new interactive analysis tool, TAPRAV. The primary components of the tool include; (i) a visualization of pupillary response aligned to the corresponding model of task execution, useful for exploring relationships between these two data sources; (ii) an interactive overview + detail metaphor, enabling rapid inspection of details while maintaining global context; (iii) synchronized playback of the video of the user’s screen interaction, providing awareness of the state of the task; and (iv) interaction supporting discovery driven analysis. Results from a user study showed that users are able to efficiently interact with the tool to analyze relationships between pupillary response and task execution. The primary contribution of our tool is that it demonstrates an effective visualization and interaction design for rapidly exploring pupillary response in relation to models of task execution, thereby reducing the analysis effort.

Existing analysis tools are not sufficient for exploring pupillary response, as the data typically needs to be explored in relation to the corresponding task's execution. To address this need, we have developed an interactive visualization tool called TAPRAV. Key components include (i) a visualization of the pupillary response aligned to the model of task execution, useful for making sense of the overall data set; (ii) an interactive overview+detail metaphor, enabling rapid inspection of details; (iii) synchronization with the video of screen interaction, providing awareness of the state of the task; and (iv) interaction supporting discovery driven analysis.

Productive collaboration in a multiple-device environment (MDE) requires an effective interface for efficiently managing applications among devices. Though many interfaces exist, there is little empirical understanding of how they affect collaboration. This paper reports results from a user study comparing how well three classes of interfaces; textual, map, and iconic, support application management during realistic, collaborative activities in an MDE. From empirical results, observations, and an analysis of how users interacted with each interface, we produced a set of design lessons for improving management interfaces. The lessons were demonstrated within the iconic interface, but they are just as applicable to other interfaces. This work contributes further understanding of how to design effective management interfaces for MDEs.

Iconic interfaces offer a promising interaction metaphor for effectively managing applications in multi-device environments. However, current implementations scale poorly for even a modest number of applications and do not allow users to maintain adequate awareness of the workspace. To overcome these limitations, we have designed new interaction techniques and prototyped them within a new iconic interface. Our interface uses zooming and animation-based interactions to improve scalability and uses application icons and portal views with real-time updates to enhance awareness. Results from a user study confirm the efficacy of these techniques. These techniques can be used to improve the broader class of iconic and portal-based interfaces.

A challenge in building interruption reasoning systems is to compute an accurate cost of interruption (COI). Prior work has used interface events and other cues to predict COI, but ignore characteristics related to the structure of a task. This work investigates how well characteristics of task structure can predict COI, as objectively measured by resumption lag. In an experiment, users were interrupted during task execution at various boundaries to collect a large sample of resumption lag values. Statistical methods were employed to create a parsimonious model that uses characteristics of task structure to predict COI. A subsequent experiment with different tasks showed that the model can predict COI with reasonably high accuracy. Our model can be expediently applied to many goal-directed tasks, allowing systems to make more effective decisions about when to interrupt.

Schools are increasingly integrating character education to facilitate improved moral thinking and pro social behavior among students. An effective method for delivering character education is problem solving moral and social situations represented visually as animated vignettes. However, schools are rarely able to use animated vignettes since existing tools do not allow them to be easily created and having them created externally is overly expensive. In this paper, we describe the design, use, and evaluation of a computational tool that enables students to construct their own animated vignettes. By building, sharing, and responding to vignettes, students become engaged in problem solving moral and social situations. Evaluations showed that users are able to build meaningful vignettes, our tool is easy to learn and fun to use, and our tool’s multimedia features are often used and well-liked. Educators can download and use our tool while researchers can draw upon our design rationale and lessons learned when building similar tools.

To contribute to systems that reason about human attention, our work empirically demonstrates how a user's mental workload changes during task execution. We conducted a study where users performed interactive, hierarchical tasks while mental workload was measured through the use of pupil size. Results show that (i) different types of subtasks impose different mental workload, (ii) workload decreases at subtask boundaries, (iii) workload decreases more at boundaries higher in a task model and less at boundaries lower in the model, (iv) workload changes among subtask boundaries within the same level of a task model, and (v) effective understanding of why changes in workload occur requires that the measure be tightly coupled to a validated task model. From the results, we show how to map mental workload onto a computational Index of Opportunity that systems can use to better reason about human attention.

User attention is a scarce resource, and users are susceptible to interruption overload. Systems do not reason about the effects of interrupting a user during a task sequence. In this study, we measure effects of interrupting a user at different moments within task execution in terms of task performance, emotional state, and social attribution. Task models were developed using event perception techniques, and the resulting models were used to identify interruption timings based on a user's predicted cognitive load. Our results show that different interruption moments have different impacts on user emotional state and positive social attribution, and suggest that a system could enable a user to maintain a high level of awareness while mitigating the disruptive effects of interruption. We discuss implications of these results for the design of an attention manager.

DEMAIS is an informal design tool that we claim helps a multimedia designer explore and communicate temporal and interactive (behavioral) design ideas better than existing tools. This paper seeks to empirically validate our claim. We report on an evaluation comparing DEMAIS to pencil and paper and Authorware for the exploration and communication of behavior in early multimedia design. The main results are that (i) DEMAIS was better than Authorware for both exploring and communicating behavior, (ii) DEMAIS was better than pencil and paper for communicating behavior, and (iii) DEMAIS was able to capture most of a designer's behavioral design ideas. Our results show that DEMAIS bridges the early investment/communication gap that exists among current multimedia design tools.

To create an innovative interactive multimedia application, a multimedia designer needs to rapidly explore numerous behavioral design ideas early in the design process, as creating innovative behavior is the cornerstone of creating innovative multimedia. Current tools and techniques do not support a designer's need for early behavior exploration, limiting her ability to rapidly explore and effectively communicate behavioral design ideas. To address this need, we have developed a sketch-based, interactive multimedia storyboard tool that uses a designer's ink strokes and textual annotations as an input design vocabulary. By operationalizing this vocabulary, the tool transforms an otherwise static sketch into a working example. The behavioral sketch can be quickly edited using gestures and an expressive visual language. By enabling a designer to explore and communicate behavioral design ideas using working examples early in the design process, our tool facilitates the creation of a more effective, compelling, and entertaining multimedia application.

As users continue offloading more control and responsibility to the computer, coordinating the asynchronous interactions between the user and computer is becoming increasingly important. Without proper coordination, an application attempting to gain the user's attention risks interrupting the user in the midst of performing another task. To justify why an application should avoid interrupting the user whenever possible, we designed an experiment measuring the disruptive effect of an interruption on a user's task performance. The experiment utilized six Web-based task categories and two categories of interruption tasks. The results of the experiment demonstrate that: (i) a user performs slower on an interrupted task than a non-interrupted task, (ii) the disruptive effect of an interruption differs as a function of the task category, and (iii) different interruption tasks cause similar disruptive effects on task performance. These results empirically validate the need to better coordinate user interactions among applications that are competing for the user's attention.

Creating innovative interactive multimedia presentations requires a great deal of time, skill, and effort. We have developed a multimedia synchronization toolkit, called Nsync (pronounced 'in-sync'), to address the complicated issues inherent in designing flexible, interactive multimedia presentations. The toolkit consists of two major components, a declarative synchronization definition language and a run-time presentation management system. The synchronization definition language supports the specification of synchronous interaction, asynchronous interaction, and fine-grained relationships among multiple media objects. The language also supports the combination of interactive behavior and fine-grained relationships through the use of conjunctive and disjunctive operators. We illustrate the use of the language through several examples. We argue that with the inclusion of asynchronous interaction, pre-computed playout schedules are too inflexible, thus we have developed a new presentat...