AI tools are increasingly deployed in community contexts. However, datasets used to evaluate AI are typically created by developers and annotators outside a given community, which can yield misleading conclusions about AI performance. How might we empower communities to drive the intentional design and curation of evaluation datasets for AI that impacts them? We investigate this question on Wikipedia, an online community with multiple AI-based content moderation tools deployed. We introduce Wikibench, a system that enables communities to collaboratively curate AI evaluation datasets, while navigating ambiguities and differences in perspective through discussion. A field study on Wikipedia shows that datasets curated using Wikibench can effectively capture community consensus, disagreement, and uncertainty. Furthermore, study participants used Wikibench to shape the overall data curation process, including refining label definitions, determining data inclusion criteria, and authoring data statements. Based on our findings, we propose future directions for systems that support community-driven data curation.

In this exploratory experience sampling method (ESM) research, we examined the perceptions of 74 smartphone users regarding the opportuneness of moments for proceeding through a four-stage notification-response process: the phone generating an alert (Alert), the user roughly glancing at the notification (Glance), engaging with it (Engage), and acting on it (Act). We investigated how the moments perceived as opportune for each of the four stages related to users' self-reported values of 20 contextual factors, and how these factors influenced users' perceived opportuneness of the moments for each stage. Our results reveal that Alert and Glance stages were perceived as more distinct, with Alert being influenced by social-environmental related factors and Glance characterized by a lower threshold for what constitutes an opportune moment. The final two stages - Engage and Act - were the most similar to each other. The findings also indicated how the influence of contextual factors on perceived opportuneness of the moments varied across factors, notification types, stages, and how such variation was manifested in the likelihood, valence, and magnitude of their overall influence.

The popularity of task-oriented chatbots is constantly growing, but smooth conversational progress with them remains profoundly challenging. In recent years, researchers have argued that chatbot systems should include guidance for users on how to converse with them. Nevertheless, empirical evidence about what to place in such guidance, and when to deliver it, has been lacking. Using a mixed-methods approach that integrates results from a between-subjects experiment and a reflection session, this paper compares the effectiveness of eight combinations of two guidance types (example-based and rule-based) at four guidance timings (service-onboarding, task-intro, after-failure, and upon-request), as measured by users’ task performance, improvement on subsequent tasks, and subjective experience. It establishes that each guidance type and timing has particular strengths and weaknesses, thus that each type/timing combination has a unique impact on performance metrics, learning outcomes, and user experience. On that basis, it presents guidance-design recommendations for future task-oriented chatbots.