Learnersourcing, an educational approach that positions students as active contributors rather than passive consumers, offers a scalable approach to co-creating instructional resources while engaging students in authentic problem-solving. However, it faces a fundamental tension: effective “learning” requires scaffolding that minimizes extraneous cognitive load and focuses attention on reasoning, while effective “sourcing” requires structure, completeness, and standardization to ensure student-generated content can be reused. These competing goals create a tradeoff: students either learn but produce content that is difficult to reuse, or generate usable resources but receive limited learning benefit. We propose a new AI-assisted learnersourcing paradigm to address this tension. By assigning collaborative roles to both learners and AI, the approach enables students to focus on cognitively meaningful sub-tasks that foster “learning”, while large language models (LLMs) handle mechanical and procedural sub-tasks for “sourcing”. Guided by user-centered design principles, we implement this workflow in ExPeerience, a system that scaffolds students in co-creating contextualized worked-out examples for database programming. Within ExPeerience, the AI serves as a collaborator for ideation, a co-creator of artifacts, and an evaluator of students’ inputs. Our evaluation with 24 participants showed that structuring AI into distinct collaborative roles improves learning engagement while producing high-quality student-generated content. Compared to a baseline using the Gemini chatbot, ExPeerience users created SQL problems in more diverse and personally meaningful contexts. They actively evaluated, edited, and refined AI-generated components, and most authored their own SQL solutions, whereas baseline participants largely accepted AI outputs without modification and did not attempt to solve the problem. Overall, ExPeerience produced more contextualized, varied, and thoughtfully constructed worked-out examples. These findings demonstrate the potential of AI-assisted learnersourcing as a paradigm to balance learning and sourcing goals. We also draw design implications for future AI-assisted learnersourcing systems that aim to produce reusable, high-quality learner-generated content while promoting educational value.

The design of user-generated content (UGC) platforms poses challenges in comprehensively addressing the ethical dimensions of recommendation algorithms and applying human-centered methods for their evaluation. This article presents a literature review of 97 studies on UGC algorithms (UGCAlgos) that incorporate human factors and user experience considerations to investigate the ethical issues explored in human-computer interaction (HCI) research. Our review identifies key themes in the ethical considerations surrounding UGCAlgos and the user modeling methods employed. We examine how common ethical concerns in recommender systems, such as content appropriateness, privacy, user engagement, transparency, fairness, and diversity, are studied and contextualized within UGC platforms. Furthermore, we summarize how these concerns are addressed through user modeling approaches, including data characterization, user context, user outsmarting, interface design, user beliefs, and community and societal impacts.

Transferring programming skills learned in the classroom to diverse real-world scenarios is both essential and challenging in computing education. This experience report describes an approach to facilitate learning transfer by fostering adaptive expertise. Students were engaged in co-creating contextualized worked-out examples, including step-by-step solutions. Through three homework assignments in a Spring 2023 database programming course, we observed substantial improvements, where students generated detailed and accurate solutions and enriched their problem-solving contexts from simple phrases to detailed stories, drawn from 17 real-life scenarios. Our results also suggest that the peer assessment process cultivated a supportive learning environment and fostered adaptive expertise. We discuss the lessons learned and draw pedagogical implications for integrating student-generated contextualized materials in other programming courses.

Missing data is a pervasive issue in real-world analytics, stemming from a multitude of factors (e.g., device malfunctions and network disruptions), making it a ubiquitous challenge in many domains. Misperception of missing data impacts decision-making and causes severe consequences. To mitigate risks from missing data and facilitate proper handling, computing methods (e.g., imputation) have been studied, which often culminate in the visual representation of data for analysts to further check. Yet, the influence of these computed representations on user judgment regarding missing data remains unclear. To study potential influencing factors and their impact on user judgment, we conducted a crowdsourcing study. We controlled 4 factors: the distribution, imputation, and visualization of missing data, and the prior knowledge of data. We compared users’ estimations of missing data with computed imputations under different combinations of these factors. Our results offer useful guidance for visualizing missing data and their imputations, which informs future studies on developing trustworthy computing methods for visual analysis of missing data.

Motivations : Recent research has emerged on generally how to improve AI products’ human-AI interaction (HAI) user experience (UX), but relatively little is known about HAI-UX inclusivity. For example, what kinds of users are supported, and who are left out? What product changes would make it more inclusive? Objectives : To help fill this gap, we present an approach to measuring what kinds of diverse users an AI product leaves out and how to act upon that knowledge. To bring actionability to the results, the approach focuses on users’ problem-solving diversity. Thus, our specific objectives were (1) to show how the measure can reveal which participants with diverse problem-solving styles were left behind in a set of AI products and (2) to relate participants’ problem-solving diversity to their demographic diversity, specifically gender and age. Methods : We performed 18 experiments, discarding two that failed manipulation checks. Each experiment was a 2 \(\times\) 2 factorial experiment with online participants, comparing two AI products: one deliberately violating 1 of 18 HAI guidelines and the other applying the same guideline. For our first objective, we used our measure to analyze how much each AI product gained/lost HAI-UX inclusivity compared to its counterpart, where inclusivity meant supportiveness to participants with particular problem-solving styles. For our second objective, we analyzed how participants’ problem-solving styles aligned with their gender identities and ages. Results and Implications : Participants’ diverse problem-solving styles revealed six types of inclusivity results: (1) the AI products that followed an HAI guideline were almost always more inclusive across diversity of problem-solving styles than the products that did not follow that guideline—but “who” got most of the inclusivity varied widely by guideline and by problem-solving style; (2) when an AI product had risk implications, four variables’ values varied in tandem: participants’ feelings of control, their (lack of) suspicion, their trust in the product, and their certainty while using the product; (3) the more control an AI product offered users, the more inclusive it was; (4) whether an AI product was learning from “my” data or other people’s affected how inclusive that product was; (5) participants’ problem-solving styles skewed differently by gender and age group; and (6) almost all of the results suggested actions that HAI practitioners could take to improve their products’ inclusivity further. Together, these results suggest that a key to improving the demographic inclusivity of an AI product (e.g., across a wide range of genders, ages) can often be obtained by improving the product’s support of diverse problem-solving styles.

This work contributes a research protocol for evaluating human-AI interaction in the context of specific AI products. The research protocol enables UX and HCI researchers to assess different human-AI interaction solutions and validate design decisions before investing in engineering. We present a detailed account of the research protocol and demonstrate its use by employing it to study an existing set of human-AI interaction guidelines. We used factorial surveys with a 2 × 2 mixed design to compare user perceptions when a guideline is applied versus violated, under conditions of optimal versus sub-optimal AI performance. The results provided both qualitative and quantitative insights into the UX impact of each guideline. These insights can support creators of user-facing AI systems in their nuanced prioritization and application of the guidelines.

Due to the scale of data and the complexity of analysis tasks, insight discovery often requires coordinating multiple visualizations (views), with each view displaying different parts of data or the same data from different perspectives. For example, to analyze car sales records, a marketing analyst uses a line chart to visualize the trend of car sales, a scatterplot to inspect the price and horsepower of different cars, and a matrix to compare the transaction amounts in types of deals. To explore related information across multiple views, current visual analysis tools heavily rely on brushing and linking techniques, which may require a significant amount of user effort (e.g., many trial-and-error attempts). There may be other efficient and effective ways of displaying cross-view data relationships to support data analysis with multiple views, but currently there are no guidelines to address this design challenge. In this article, we present systematic design considerations for visualizing cross-view data relationships, which leverages descriptive aspects of relationships and usable visual context of multi-view visualizations. We discuss pros and cons of different designs for showing cross-view data relationships, and provide a set of recommendations for helping practitioners make design decisions.

Major breaches of sensitive company data, as for Facebook's 50 million user accounts in 2018 or Equifax's 143 million user accounts in 2017, are showing the limitations of reactive data security technologies. Companies and government organizations are turning to proactive data security technologies that secure sensitive data at source. However, data security analysts still face two fundamental challenges in data protection decisions: 1) the information overload from the growing number of data repositories and protection techniques to consider; 2) the optimization of protection plans given the current goals and available resources in the organization. In this work, we propose an intelligent user interface for security analysts that recommends what data to protect, visualizes simulated protection impact, and helps build protection plans. In a domain with limited access to expert users and practices, we elicited user requirements from security analysts in industry and modeled data risks based on architectural and conceptual attributes. Our preliminary evaluation suggests that the design improves the understanding and trust of the recommended protections and helps convert risk information in protection plans.

Crowdsourced sensemaking has shown great potential for enabling scalable analysis of complex data sets, from planning trips, to designing products, to solving crimes. Yet, most crowd sensemaking approaches still require expert intervention because of worker errors and bottlenecks that would otherwise harm the output quality. Mitigating these errors and bottlenecks would significantly reduce the burden on experts, yet little is known about the types of mistakes crowds make with sensemaking micro-tasks and how they propagate in the sensemaking loop. In this paper, we conduct a series of studies with 325 crowd workers using a crowd sensemaking pipeline to solve a fictional terrorist plot, focusing on understanding why errors and bottlenecks happen and how they propagate. We classify types of crowd errors and show how the amount and quality of input data influence worker performance. We conclude by suggesting design recommendations for integrated crowdsourcing systems and speculating how a complementary top-down path of the pipeline could refine crowd analyses.

The increasing volume of text data is challenging the cognitive capabilities of expert analysts. Machine learning and crowdsourcing present new opportunities for large-scale sensemaking, but we must overcome the challenge of modeling the overall process so that many distributed agents can contribute to suitable components asynchronously and meaningfully. In this paper, we explore how to crowdsource the sensemaking process via a pipeline of modularized steps connected by clearly defined inputs and outputs. Our pipeline restructures and partitions information into "context slices" for individual workers. We implemented CrowdIA, a software platform to enable unsupervised crowd sensemaking using our pipeline. With CrowdIA, crowds successfully solved two mysteries, and were one step away from solving the third. The crowd's intermediate results revealed their reasoning process and provided evidence that justifies their conclusions. We suggest broader possibilities to optimize each component, as well as to evaluate and refine previous intermediate analyses to improve the final result.