Specifying model behavior is challenging—especially when the desired behavior is unpopular relative to the model’s training data. Reversing the influence of massive training corpora is both time-consuming and costly, and such interventions are typically inaccessible to end users. While Large Language Models (LLMs) make it easier to write instructions using natural language, specifying unpopular behaviors remains a difficult task.

Contributors to open source software packages often describe feeling discouraged by the lack of positive feedback from users. This paper describes a technology probe, Hug Reports, that provides users a communication affordance within their code editors, through which users can convey appreciation to contributors of packages they use. In our field study, 18 users interacted with the probe for 3 weeks, resulting in messages of appreciation to 550 contributors, 26 of whom participated in subsequent research. Our findings show how locating a communication affordance within the code editor, and allowing users to express appreciation in terms of the abstractions they are exposed to (packages, modules, functions), can support exchanges of appreciation that are meaningful to users and contributors. Findings also revealed the moments in which users expressed appreciation, the two meanings that appreciation took on -- as a measure of utility and as an act of expressive communication -- and how contributors' reactions to appreciation were influenced by their perceived level of contribution. Based on these findings, we discuss opportunities and challenges for designing appreciation systems for open source in particular, and peer production communities more generally.

Prompt-based interfaces for Large Language Models (LLMs) have made prototyping and building AI-powered applications easier than ever before. However, identifying potential harms that may arise from AI applications remains a challenge, particularly during prompt-based prototyping. To address this, we present Farsight, a novel in situ interactive tool that helps people identify potential harms from the AI applications they are prototyping. Based on a user’s prompt, Farsight highlights news articles about relevant AI incidents and allows users to explore and edit LLM-generated use cases, stakeholders, and harms. We report design insights from a co-design study with 10 AI prototypers and findings from a user study with 42 AI prototypers. After using Farsight, AI prototypers in our user study are better able to independently identify potential harms associated with a prompt and find our tool more useful and usable than existing resources. Their qualitative feedback also highlights that Farsight encourages them to focus on end-users and think beyond immediate harms. We discuss these findings and reflect on their implications for designing AI prototyping experiences that meaningfully engage with AI harms. Farsight is publicly accessible at: https://pair-code.github.io/farsight.

Creative entrepreneurs rely on online platforms to build community and overcome isolated work conditions. However, because of frequent attempts by larger brands to use their work without permission, creative entrepreneurs constrain their use of social platforms to safeguard their intellectual property. In this paper, we describe a multi-year partnership with a feminist makerspace to build a social platform, called Peerdea, that centered creative entrepreneurs' needs such that online feedback, information exchange, goal setting, and accountability were more readily available to them. Through an iterative, community-collaborative approach with 46 creative entrepreneurs, we report on the kinds of peer support entrepreneurs sought on Peerdea such as feedback on in-progress and unpolished work. We argue that by aligning Peerdea's design with the makerspace's community of practice, Peerdea leveraged the relationship and trust building that occurs more readily in person for entrepreneurs. In addition, we highlight the role of a community leader who actively managed the relationships between researchers and entrepreneurs, surfaced research failures and championed successes, and provided critical mediation for co-design when participants' livelihoods were implicated.

Labor shortages have shaped many industries over the past several years, with hospitality experiencing one of the largest rates of attrition. Workers are leaving their jobs for a variety of reasons, ranging from burnout and work intensification to a lack of meaningful employment. While some literature maintains that labor-replacing automation is poised to bridge the shortages, we argue there is an opportunity for technology design to instead improve job quality and retention. Drawing on interviews with unionized guest room attendants, we report on workers’ perceptions of a widely-used algorithmic room assignment system. We then present worker-generated design ideas that adapt this system toward supporting three key facets of wellbeing: self-efficacy, transparency, and workload. We argue for the need to consider these facets of wellbeing through design across the service landscape, particularly as HCI attends to the impacts of AI and automation on frontline work.

This paper examines the art practices, artwork, and motivations of prolific users of the latest generation of text-to-image models. Through interviews, observations, and a user survey, we present a sampling of the artistic styles and describe the developed community of practice around generative AI. We find that: 1) artists hold the text prompt and the resulting image can be considered collectively as a form of artistic expression (prompts as art), and 2) prompt templates (prompts with “slots” for others to fill in with their own words) are developed to create generative art styles. We discover that the value placed by this community on unique outputs leads to artists seeking specialized vocabulary to produce distinctive art pieces (e.g., by reading architectural blogs to find phrases to describe images). We also find that some artists use “glitches” in the model that can be turned into artistic styles of their own right. From these findings, we outline specific implications for design regarding future prompting and image editing options.

Initiating conversations with new people at work is often intimidating because of uncertainty about their interests. People worry others may reject their attempts to initiate conversation or that others may not enjoy the conversation. We introduce a new system, Nooks, built on Slack, that reduces fear of social evaluation by enabling individuals to initiate any conversation as a nook—a conversation room that identifies its topic, but not its creator. Automatically convening others interested in the nook, Nooks further reduces fears of social evaluation by guaranteeing individuals in advance that others they are about to interact with are interested in the conversation. In a multi-month deployment with participants in a summer research program, Nooks provided participants with non-threatening and inclusive interaction opportunities, and ambient awareness, leading to new interactions online and offline. Our results demonstrate how intentionally designed social spaces can reduce fears of social evaluation and catalyze new workplace connections.

Online platforms for freelancing allow teams performing complex work to be assembled in a matter of minutes and dispersed nearly as quickly. With such short time frames, ad hoc and virtual teams have few opportunities to learn strategies and effective team practices to work with their colleagues. Without such practices, teams are prone to work sub-optimally and lack direction. One key challenge in virtual teams discovering effective team practices is that because the practices ought to involve situated knowledge, it takes time to coalesce, as team members learn about each other over time. This work introduces Retrospector, that ad hoc teams can use to reflect collaboratively and reinforce effective team practices. Our interface accelerates the discovery of practices in situ and then guides them in reinforcing and applying these practices to future tasks. We conducted a between-subjects experiment (N=75) to assess our design with crowdworkers from the Amazon Mechanical Turk platform. This randomized controlled experiment showed that teams using our system for approximately six minutes of collaborative reflection were able to discover effective practices more successfully and had significantly improved team performance and viability. These results indicate that deliberate support for improving team practices can improve outcomes even through very short interaction. We conclude with design implications and opportunities for future work.

Recent investments in automation and AI are reshaping the hospitality sector. Driven by social and economic forces affecting service delivery, these new technologies have transformed the labor that acts as the backbone to the industry-namely frontline service work performed by housekeepers, front desk staff, line cooks and others. We describe the context for recent technological adoption, with particular emphasis on algorithmic management applications. Through this work, we identify gaps in existing literature and highlight areas in need of further research in the domains of worker-centered technology development. Our analysis highlights how technologies such as algorithmic management shape roles and tasks in the high-touch service sector. We outline how harms produced through automation are often due to a lack of attention to non-management stakeholders. We then describe an opportunity space for researchers and practitioners to elicit worker participation at all stages of technology adoption, and offer methods for centering workers, increasing transparency, and accounting for the context of use through holistic implementation and training strategies.

Smartphone-based trading apps such as Robinhood and Webull have risen in popularity over the past few years. These apps allow investors with little prior investing experience easy and inexpensive (often commission-free) access to trading stocks, options, and other securities. However, non-expert investors using these apps often make poor investing decisions due to behavioral factors. In particular, such investors 1) trade more frequently leading to short-term speculation rather than reaping the long-term benefits of their investments, 2) make investing decisions based on emotion rather than economic or financial considerations, and 3) under-diversify their portfolio, leading to unnecessarily large risks. Together, these actions reduce their investor returns, and sometimes lead to devastating losses that were avoidable. This paper introduces Robinhood’s Forest, an idle game that helps non-expert investors improve investing behavior. Unlike interactive digital games that emphasize interactivity, idle games are designed for interpassivity. Idle games are based on the premise that “waiting is playing” and players can derive pleasure by repeating simple actions or automating them. As such, Robinhood’s Forest 1) provides recurring gratification from limiting investing actions and encouraging long-term investing, 2) abstracts representations of investments to reduce overreaction to market news and social pressure, and 3) encourages diversification by using strong visual metaphors. We conducted two small-scale lab studies that demonstrate that Robinhood’s Forest reduced participants’ desire for frequent trading and encouraged them to diversify their portfolios. At the same time, participants in our studies still desired functionality like performance visualizations and market news updates that allowed them to keep up with the market. Based on these findings, we also discuss design implications for other interactive systems that emphasize non-interaction.

When members of ad-hoc virtual teams need to collectively ideate or deliberate, they often fail to engage with each others' perspectives in a constructive manner. At best, this leads to sub-optimal outcomes, and, at worst, it can cause conflicts that lead to teams not wanting to continue working together. Prior work has attempted to facilitate constructive communication by highlighting problematic communication patterns and nudging teams to alter their interaction norms. However, these approaches achieve limited success because they fail to acknowledge two social barriers: (1) it is hard to reset team norms mid-interaction, and (2) corrective nudges have limited utility unless team members believe it is safe to voice their opinion and that their opinion will be heard. This paper introduces Empathosphere, a chat-embedded intervention to mitigate these barriers and foster constructive communication in teams. To mitigate the first barrier, Empathosphere leverages the known benefits of "experimental spaces" in dampening existing norms and creating a climate conducive to change. Empathosphere instantiates this "space'' as a separate communication channel in a team's workspace. To mitigate the second barrier, Empathosphere harnesses the benefits of perspective-taking to cultivate a group climate that promotes a norm of members speaking up and engaging with each other. Empathosphere achieves this by orchestrating authentic socio-emotional exchanges designed to induce perspective-taking. A controlled study ($N=110$) compared Empathosphere to an alternate intervention strategy of prompting teams to reflect on their team experience. We found that Empathosphere led to higher work satisfaction, encouraged more open communication and feedback within teams, and boosted teams' desire to continue working together. This work demonstrates that "experimental spaces," particularly those that integrate methods of encouraging perspective-taking, can be a powerful means of improving communication in virtual teams.

Smartphone apps such as Robinhood and Public that promise to “democratize investing” have risen in popularity over the past few years. These apps allow retail investors, who often possess little prior investing experience, to trade stocks, options, and other securities easily and inexpensively, often commission-free. It seems plausible that the interaction patterns of these new apps may significantly influence trading behaviors of their users. But so far, there is little formal design guidance on how such apps should be designed. This paper introduces a set of design guidelines for encouraging healthy investing behaviors by drawing on three bodies of related work: 1) findings from finance and economics literature on healthy investment practices, 2) the dual process theory from behavioral sciences, and 3) design metaphors used in interfaces with uncertain rewards. Using these guidelines, we qualitatively analyze the user interfaces of some popular investment platforms. Our analysis reveal that, unfortunately, popular trading apps generally do not follow design patterns that encourage healthier trading behaviors. We discuss design implications and opportunities for future design.

Voice assistants are fundamentally changing the way we access information. However, voice assistants still leverage little about the web beyond simple search results. We introduce Firefox Voice, a novel voice assistant built on the open web ecosystem with an aim to expand access to information available via voice. Firefox Voice is a browser extension that enables users to use their voice to perform actions such as setting timers, navigating the web, and reading a webpage’s content aloud. Through an iterative development process and use by over 12,000 active users, we find that users see voice as a way to accomplish certain browsing tasks efficiently, but struggle with discovering functionality and frequently discontinue use. We conclude by describing how Firefox Voice enables the development of novel, open web-powered voice-driven experiences.

Social media can be an effective source of feedback on open-ended work, such as product design. Unlike large businesses with entire teams dedicated to "social," little is understood about how small business owners-constrained both in personnel and resources-leverage the benefits of direct, informal communication channels afforded by social media. Drawing on a series of design workshops and interviews with 26 small business owners at a local feminist makerspace, we report on the unique challenges small business owners experience when seeking feedback on open-ended work via social media. We found participants carefully balanced large-scale access to diverse audiences with attempts to receive reliable feedback, and they often targeted audiences narrowly to reinstate control and build trust. In addition, the small business owners in our workshop idealized building authentic relationships with their social audiences to create collectively. To do so successfully, participants detailed the extensive behind-the-scenes work required of them such as navigating blurred personal and business identities and the self-regulation necessary to continuously stay engaged and not internalize discouraging feedback.

How might the capabilities of voice assistants several decades in the future shape human society? To anticipate the space of possible futures for voice assistants, we asked 149 participants to each complete a story based on a brief story stem set in the year 2050 in one of five different contexts: the home, doctor's office, school, workplace, and public transit. Story completion as a method elicits participants' visions of possible futures, unconstrained by their understanding of current technological capabilities, but still reflective of current sociocultural values. Through a thematic analysis, we find these stories reveal the extremes of the capabilities and concerns of today's voice assistants---and artificial intelligence---such as improving efficiency and offering instantaneous support, but also replacing human jobs, eroding human agency, and causing harm through malfunction. We conclude by discussing how these speculative visions might inform and inspire the design of voice assistants and other artificial intelligence.

Expert crowdsourcing (e.g., Upwork.com) provides promising benefits such as productivity improvements for employers, and flexible working arrangements for workers. Yet to realize these benefits, a key persistent challenge is effective hiring at scale. Current approaches, such as reputation systems and standardized competency tests, develop weaknesses such as score inflation over time, thus degrading market quality. This paper presents HirePeer, a novel alternative approach to hiring at scale that leverages peer assessment to elicit honest assessments of fellow workers' job application materials, which it then aggregates using an impartial ranking algorithm. This paper reports on three studies that investigate both the costs and the benefits to workers and employers of impartial peer-assessed hiring. We find, to solicit honest assessments, algorithms must be communicated in terms of their impartial effects. Second, in practice, peer assessment is highly accurate, and impartial rank aggregation algorithms incur a small accuracy cost for their impartiality guarantee. Third, workers report finding peer-assessed hiring useful for receiving targeted feedback on their job materials.

Conceptual diagrams are used extensively to understand abstract relationships, explain complex ideas, and solve difficult problems. To illustrate concepts effectively, experts find appropriate visual representations and translate concepts into concrete shapes. This translation step is not supported explicitly by current diagramming tools. This paper investigates how domain experts create conceptual diagrams via semi-structured interviews with 18 participants from diverse backgrounds. Our participants create, adapt, and reuse visual representations using both sketches and digital tools. However, they had trouble using current diagramming tools to transition from sketches and reuse components from earlier diagrams. Our participants also expressed frustration with the slow feedback cycles and barriers to automation of their tools. Based on these results, we suggest four opportunities of diagramming tools — exploration support, representation salience, live engagement, and vocabulary correspondence — that together enable a natural diagramming experience. Finally, we discuss possibilities to leverage recent research advances to develop natural diagramming tools.

Online freelancing is growing rapidly. However, despite this rapid growth, we have a limited understanding of online freelancers' long-term experiences and evolution, or how online freelancing influences freelancers' broader career goals. To address this gap, we interviewed a set of online freelancers at two time periods, two and a half years apart. We found that long-term engagement with online freelancing involves a unique set of financial, emotional, relational, and reputational burdens that represent the overhead of maintaining an online freelancing career. We found that this overhead influenced online freelancers' participation and perceptions of online freelancing over time, as well as the strategies some freelancers employed to manage their careers. Our findings further highlight how online freelance platforms can afford unique career development opportunities over a longer period of time, including career exploration and transition, entrepreneurial training and reputation, and skills transfer. Based on our findings, we present policy and design implications to increase the sustainability and accessibility of online freelancing.

Attracting and retaining newcomers is critical and challenging for online production communities such as Wikipedia, both because volunteers need specialized training and are likely to leave before being integrated into the community. In response to these challenges, the Wikimedia Foundation started the Wiki Education Project (Wiki Ed), an online program in which college students edit Wikipedia articles as class assignments. The Wiki Ed program incorporates many components of institutional socialization, a process many conventional organizations successfully use to integrate new employees through formalized on-boarding practices. Research has not adequately investigated whether Wiki Ed and similar programs are effective ways to integrate volunteers in online communities, and, if so, the mechanisms involved. This paper evaluates the Wiki Ed program by comparing 16,819 student editors in 770 Wiki Ed classes with new editors who joined Wikipedia in the conventional way. The evaluation shows that the Wiki Ed students did more work, improved articles more, and were more committed to Wikipedia. For example, compared to new editors who joined Wikipedia in the conventional way they were twice as likely to still be editing Wikipedia a year after their Wiki Ed class was finished. Further, students in classrooms that encouraged joint activity, a key component of institutional socialization, produced better quality work than those in classrooms where students worked independently. These findings are consistent with an interpretation that the Wiki Ed program was successful because it incorporated elements of institutionalized socialization.

This paper investigates whether voice assistants can play a useful role in the specialized work-life of the knowledge worker (in a biology lab). It is motivated both by promising advances in voice-input technology, and a long-standing vision in the community to augment scientific processes with voice-based agents. Through a reflection on our design process and a limited but fully functional prototype, Vitro, we find that scientists wanted a voice-enabled device that acted not a lab assistant, but lab equipment. Second, we discovered that such a device would need to be deeply embedded in the physical and social space in which it served scientists. Finally, we discovered that scientists preferred a device that supported their practice of "careful deviation" from protocols in their lab work. Through this research, we contribute implications for the design of voice-enabled systems in workplace settings.

How do we design technology for learning at scale? Based on an examination of a large number of influential systems for learning at scale, I argue that designing for scale is not an amorphous design undertaking. Instead, it builds on two distinct perspectives on scale. Systems built with a scaling through efficiency perspective make learning more efficient and allow the same number of instructors to help a much larger set of learners. Systems with a scaling through empowerment perspective empower a larger number of people to assist learners effectively. I outline how these simple differences in design perspective lead to large differences in design concerns, techniques, and evaluation criteria. Articulating prevalent design perspectives should make overlooked design opportunities more salient, help systems designers design for scale more deliberately and understand their tradeoffs, and open up new opportunities to designers who shift their perspectives.

When a smart device talks, what should its voice sound like? Voice-enabled devices are becoming a ubiquitous presence in our everyday lives. Simultaneously, speech synthesis technology is rapidly improving, making it possible to generate increasingly varied and realistic computerized voices. Despite the flexibility and richness of expression that technology now affords, today's most common voice assistants often have female-sounding, polite, and playful voices by default. In this paper, we examine the social consequences of voice design, and introduce a simple research framework for understanding how voice affects how we perceive and interact with smart devices. Based on the foundational paradigm of computers as social actors, and informed by research in human-robot interaction, this framework demonstrates how voice design depends on a complex interplay between characteristics of the user, device, and context. Through this framework, we propose a set of guiding questions to inform future research in the space of voice design for smart devices.

We study rank aggregation algorithms that take as input the opinions of players over their peers, represented as rankings, and output a social ordering of the players (which reflects, e.g., relative contribution to a project or fit for a job). To prevent strategic behavior, these algorithms must be impartial, i.e., players should not be able to influence their own position in the output ranking. We design several randomized algorithms that are impartial and closely emulate given (non-impartial) rank aggregation rules in a rigorous sense. Experimental results further support the efficacy and practicability of our algorithms.

Peer review asks novices to take on an evaluator's role, yet novices often lack the perspective to accurately assess the quality of others' work. To help learners give feedback on their peers' work through an expert lens, we present the Juxtapeer peer review system for structured comparisons. We build on theories of learning through contrasting cases, and contribute the first systematic evaluation of comparative peer review. In a controlled experiment, 476 consenting learners across four courses submitted 1,297 submissions, 4,102 reviews, and 846 self assessments. Learners assigned to compare submissions wrote reviews and self-reflections that were longer and received higher ratings from experts than those who evaluated submissions one at a time. A second study found that a ranking of submissions derived from learners' comparisons correlates well with staff ranking. These results demonstrate that comparing algorithmically-curated pairs of submissions helps learners write better feedback.

Soil moisture is critical to agriculture, ecology, and certain natural disasters. Existing soil moisture models often fail to predict soil moisture accurately for time periods greater than a few hours. To tackle this problem, we introduce in this paper two novel models, the Naive Accumulative Representation (NAR) and the Additive Exponential Accumulative Representation (AEAR). The parameters in these models reflect hydrological redistribution processes of gravity and suction. We validate our models using soil moisture and rainfall time series data collected from a steep gradient post-wildfire site in Southern California. Data analysis is challenging, since rapid landscape change in steep, burned hillslopes is typically observed in response to even small to moderate rain events. We found that the AEAR model fits the data well for three distinct soil textures at different depths below the ground surface (at 5cm, 15cm, and 30cm). Similar strong results are demonstrated in controlled soil moisture experiments. Our recommended AEAR model has been validated as effective and useful by earth scientists, giving better forecasts than existing models for time horizons of 10 to 24 hours.

Soil moisture plays a crucial part in earth science, with impact on agriculture, ecology, hydrology, landslides, and water resources. Extremes in soil moisture, which we denote as peaks and valleys, caused by heavy rainfalls and subsequent dry weather, are very important when predicting future soil moisture or even landslides. Existing methods, like moving averages, have limitations when it comes to smoothing time series data while preserving peaks and valleys. In this work, we propose a novel method, HyperSTL, for extrema-preserving smoothing of soil moisture time series. The method optimizes an existing time series decomposition technique, Seasonal Decomposition of Time Series by Loess (STL). HyperSTL optimizes STL's control parameters, which we call hyperparameters, using an objective function over the decomposed components. We demonstrate in experiments with nine soil moisture datasets that using HyperSTL generally results in improved predictions compared to using other smoothing methods.

Many studies demonstrate that peer reviewing provides pedagogical benefits such as inspiration and developing expert vision, and changes classroom culture by encouraging reciprocity. However, much large-scale research in peer assessment has focused on MOOCs, where students have short tenures, and is unable to describe how reciprocity-oriented classroom cultures evolve over time. This short paper presents the first long-term analysis of peer reviewing with 304 students, conducted in three large physical classes in a year-long undergraduate series. Surprisingly, this analysis reveals that when students receive better reviews on their work, they write worse reviews in the future. This suggests that while students believe in the reciprocal nature of peer review, they act anti-reciprocally. Therefore, battling the emergent norm of anti-reciprocity is crucial both for system designers and practitioners who use peer assessment.

Massive online classes can benefit from peer interactions such as discussion, critique, or tutoring. However, to scaffold productive peer interactions, systems must be able to detect student behavior in interactions at scale, which is challenging when interactions occur over rich media like video. This paper introduces an imprecise yet simple browser-based conversational turn detector for video conversations. Turns are detected without accessing video or audio data. We show how this turn detector can find dominance in video-based conversations. In a case study with 1,027 students using Talkabout, a video-based discussion system for online classes, we show how detected conversational turn behavior correlates with participants' subjective experience in discussions and their final course grade.

Massive online classes are global and diverse. How can we harness this diversity to improve engagement and learning? Currently, though enrollments are high, students' interactions with each other are minimal: most are alone together. This isolation is particularly disappointing given that a global community is a major draw of online classes. This paper illustrates the potential of leveraging geographic diversity in massive online classes. We connect students from around the world through small-group video discussions. Our peer discussion system, Talkabout, has connected over 5,000 students in fourteen online classes. Three studies with 2,670 students from two classes found that globally diverse discussions boost student performance and engagement: the more geographically diverse the discussion group, the better the students performed on later quizzes. Through this work, we challenge the view that online classes are useful only when in-person classes are unavailable. Instead, we demonstrate how diverse online classrooms can create benefits that are largely unavailable in a traditional classroom.

When students work with peers, they learn more actively, build richer knowledge structures, and connect material to their lives. However, not every peer learning experience online sees successful adoption. This paper articulates and addresses three adoption challenges for global-scale peer learning. First, peer interactions struggle to bootstrap critical mass. However, class incentives can signal importance and spur initial usage. Second, online classes have limited peer visibility and awareness, so students often feel alone even when surrounded by peers. We find that highlighting interdependence and strengthening norms can mitigate this issue. Third, teachers can readily access "big" aggregate data but not "thick" contextual data that helps build intuitions, so software should guide teachers' scaffolding of peer interactions. We illustrate these challenges through studying 8,500 students' usage of two peer learning platforms, Talkabout and PeerStudio. This paper measures efficacy through sign-up and participation rates and the structure and duration of student interactions.

Rapid feedback is a core component of mastery learning, but feedback on open-ended work requires days or weeks in most classes today. This paper introduces PeerStudio, an assessment platform that leverages the large number of students' peers in online classes to enable rapid feedback on in-progress work. Students submit their draft, give rubric-based feedback on two peers' drafts, and then receive peer feedback. Students can integrate the feedback and repeat this process as often as they desire. In MOOC deployments, the median student received feedback in just twenty minutes. Rapid feedback on in-progress work improves course outcomes: in a controlled experiment, students' final grades improved when feedback was delivered quickly, but not if delayed by 24 hours. More than 3,600 students have used PeerStudio in eight classes, both massive and in-person. This research demonstrates how large classes can leverage their scale to encourage mastery through rapid feedback and revision.

Student discussions over video in massive classes allow students to explore course content, share personal experiences and get feedback on their ideas. However, such discussions frequently turn into casual conversations without focusing on the curriculum and the learning objectives. This short paper explores whether students can achieve multiple learning objectives by solving challenges collaboratively during discussions. We introduce `think-pair-share' technique for video discussions. Our pilot results, drawn from a Coursera class, suggest that participants prefer to exchange information with their peers using personal stories and connecting stories with curriculum increases participant engagement.

In the physical classroom, peer interactions motivate students and expand their perspective. We suggest that synchronous peer interaction can benefit massive online courses as well. Talkabout organizes students into video discussion groups and allows instructors to determine group composition and discussion content. Using Talkabout, students pick a discussion time that suits their schedule. The system groups the students into small video discussions based on instructor preferences such as gender or geographic balance. To date, 2,474 students in five massive online courses have used Talkabout to discuss topics ranging from prejudice to organizational theory. Talkabout discussions are diverse: in one course, the median six-person discussion group had students from four different countries. Students enjoyed discussing in these diverse groups: the average student participated for 66 minutes, twice the course requirement. Students in more geographically distributed groups also scored higher on the final, suggesting that distributed discussions have educational value.

Peer assessment helps students reflect and exposes them to different ideas. It scales assessment and allows large online classes to use open-ended assignments. However, it requires students to spend significant time grading. How can we lower this grading burden while maintaining quality? This paper integrates peer and machine grading to preserve the robustness of peer assessment and lower grading burden. In the identify-verify pattern, a grading algorithm first predicts a student grade and estimates confidence, which is used to estimate the number of peer raters required. Peers then identify key features of the answer using a rubric. Finally, other peers verify whether these feature labels were accurately applied. This pattern adjusts the number of peers that evaluate an answer based on algorithmic confidence and peer agreement. We evaluated this pattern with 1370 students in a large, online design class. With only 54% of the student grading time, the identify-verify pattern yields 80-90% of the accuracy obtained by taking the median of three peer scores, and provides more detailed feedback. A second experiment found that verification dramatically improves accuracy with more raters, with a 20% gain over the peer-median with four raters. However, verification also leads to lower initial trust in the grading system. The identify-verify pattern provides an example of how peer work and machine learning can combine to improve the learning experience.

As news reading becomes more social, how do different types of annotations affect people's selection of news articles? This paper reports on results from two experiments looking at social annotations in two different news reading contexts. The first experiment simulates a logged-out experience with annotations from strangers, a computer agent, and a branded company. Results indicate that, perhaps unsurprisingly, annotations by strangers have no persuasive effects. However, surprisingly, unknown branded companies still had a persuasive effect. The second experiment simulates a logged-in experience with annotations from friends, finding that friend annotations are both persuasive and improve user satisfaction over their article selections. In post-experiment interviews, we found that this increased satisfaction is due partly because of the context that annotations add. That is, friend annotations both help people decide what to read, and provide social context that improves engagement. Interviews also suggest subtle expertise effects. We discuss implications for design of social annotation systems and suggestions for future research.

Abstract We introduce an algorithm for automatic selection of semantically‐resonant colors to represent data (e.g., using blue for data about “oceans”, or pink for “love”). Given a set of categorical values and a target color palette, our algorithm matches each data value with a unique color. Values are mapped to colors by collecting representative images, analyzing image color distributions to determine value‐color affinity scores, and choosing an optimal assignment. Our affinity score balances the probability of a color with how well it discriminates among data values. A controlled study shows that expert‐chosen semantically‐resonant colors improve speed on chart reading tasks compared to a standard palette, and that our algorithm selects colors that lead to similar gains. A second study verifies that our algorithm effectively selects colors across a variety of data categories.

Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9% of students’ grades within 5% of the staff grade, and 65.5% within 10%. On average, students assessed their work 7% higher than staff did. Students also rated peers’ work from their own country 3.6% higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4% to 9.9%.

The cultural heritage of a region is conveyed by both tangible physical artifacts and intangible aspects in the form of stories, dance styles, rituals, etc. Hitherto, the task of creating digital representations for each of these aspects has been addressed in isolation, i.e. using specific media most suited to the artifact such as video, audio, three-dimensional (3D) models, scanning, etc. The challenge of bringing together these separate elements to create a coherent story, however, has remained unaddressed until recently. In this paper we present a unified digital framework that enables the integration of disparate representations of heritage elements into a holistic entity. Our approach results in a compelling and engaging narration that affords a unified user experience. Our solution supports both active (user-controlled explorations) and passive (watching pre-orchestrated narrations) user interactions.

Transactional Memory aims to provide a programming model that makes parallel programming easier. Hardware implementations of transactional memory (HTM) suffer from fewer overheads than implementations in software, and refinements in conflict management strategies for HTM allow for even larger improvements. In particular, lazy conflict management has been shown to deliver better performance, but it has hitherto required complex protocols and implementations.

Boosted transactions offer an attractive method that enables programmers to create larger transactions that scale well and offer deadlock-free guarantees. However, as boosted transactions get larger, they become more susceptible to conflicts and aborts. We describe a linear-time algorithm to detect transactions that cannot make progress, which transactions need to be aborted, and when. The algorithm guarantees zero false positives with minimal aborts. Our proposals, as implemented in DSTM2, increase the transactional throughput of the system, often by more than 30%.