Benchmarks play a significant role in how technology companies communicate about model capabilities and how researchers and the public understand generative AI systems. However, existing benchmarks have been criticized for their failure to adequately capture real-world usages (i.e. ecological validity) or to measure underlying concepts (i.e. construct validity). Building on approaches in HCI, we adopt a human-centered design process to address such critiques. Working within the journalism domain we engaged 23 professionals in a workshop which informed the design of a domain-oriented evaluation “cookbook”. Our workshop findings surface domain-specific challenges and tensions faced by designers in translating specific tasks to evaluation constructs, aligning metrics with domain-specific values, and balancing needs among different stakeholders when constructing evaluations. Through an instantiation of design-based approaches for benchmark creation in the journalism domain, this work not only produces an evaluation structure for journalism practitioners to experiment with, but also lays out design requirements for AI evaluations that are contextualized, value-aligned, and cultivate evaluative literacy for domain end-users.

Journalists rely on their agency—the ability to exercise independent judgment in alignment with their values—to fulfill their democratic social role. In this study, we investigate how LLM-infused writing tools reshape journalists’ agency in editorial decision making. In interviews with 20 science journalists, we presented four hypothetical LLM-infused writing tools representing a range of possible design space configurations. We find that journalists are selectively willing to cede control: they view AI that gathers information or offers feedback as supporting their efficiency by automating execution while leaving decision making intact. In contrast, they see AI that generates core ideas or drafts as a threat to their autonomy, skill development, self-fulfillment, and professional relationships. This sensitivity extends to seemingly automatable tasks such as manipulating writing voice with AI, which are seen as reducing opportunities for reflection and critical thinking. We discuss the implications of these findings for design that preserves journalistic agency in the moment, and over the long term.

Through a systematic review of design contributions in journalism, this work examines how domain-specific values shape sociotechnical systems for newswork.We illustrate the different ways in which values define design problems and act as guiding principles for solutions.For instance, the value "accountability" functions as both a design problem (how to support journalists in accountability reporting) and as a guiding principle (features to ensure that systems remain accountable to users).Our analysis reveals how ten domain values shape design choices, and how these values can support or conflict with each other in practice.Building on these findings, we then discuss how designers might position their work in relation to stakeholders: journalists, the public, and technology providers.Each of these relationships presents unique value tensions for designers to consider and balance.In this way, our work provides practical guidance for creating systems that better serve newswork, helps designers reflect on how their choices impact different stakeholders, and contributes to critical computing discourses on where values require adjudication or deeper attention.

Science and technology journalists today face challenges in finding newsworthy leads due to increased workloads, reduced resources, and expanding scientific publishing ecosystems. Given this context, we explore computational methods to aid these journalists' news discovery in terms of their agency and time-efficiency. We prototyped three computational information subsidies into an interactive tool that we used as a probe to better understand how such a tool may offer utility or more broadly shape the practices of professional science journalists. Our findings highlight central considerations around science journalists' user agency, contexts of use, and professional responsibility that such tools can influence and could account for in design. Based on this, we suggest design opportunities for enhancing and extending user agency over the longer-term; incorporating contextual, personal and collaborative notions of newsworthiness; and leveraging flexible interfaces and generative models. Overall, our findings contribute a richer view of the sociotechnical system around computational news discovery tools, and suggest ways to improve such tools to better support the practices of science journalists.

The death of Indian film star Sushant Singh Rajput at the peak of the COVID lockdown triggered chaos on the news cycle in India with a range of conspiracy theories that led to a witch hunt of sorts, and the hounding of several entertainers and public figures in the months that followed. Using data from Twitter, YouTube, and an archive of debunked misinformation stories, we examine the drivers and consequences of social media outrage in this case. We analyse these patterns from the framework of conspiracy and astroturfing and contextualize our findings to the socio-political background currently prevalent in India. Primarily, retweet rates on Twitter suggest that commentators benefited from talking about the case, which got higher engagement than other topics. Moreover, we report evidence of political hands in the way the discourse has shaped online, but more importantly that the story bears warnings for the shape and impact of witch-hunts in the backdrop of a fractured media environment. In conclusion, we consider the effects of Rajput’s outsider status as a small-town implant in the film industry within the broader narrative of systemic injustice, as well as the gendered aspects of mob justice that have taken aim at his former partner in the months since.

Celebrity influencers are increasingly central to political discourse as they engage in, and get engaged with, on matters of electoral importance. In this paper, using Twitter data from 1432 sportspersons and entertainers and their engagement with the 1000 of the most followed ruling party and opposition politicians from India, we propose a new method to measure partisanship of celebrities along different modes of engagement. Our examination of polarization, through topical and retweet analyses, shows patterns related to both party incumbency and the level of internal organization. We find that the ruling BJP has been more effective than the opposition, the INC, in organized outreach to celebrities, by eschewing explicit party-based partisanship, and instead employing non-partisan narrative techniques, such as maintaining nationalism as the central theme in tweets. We find that while entertainers are equally engaged by both the ruling and opposition parties, sportspersons, who often enjoy a nationalist appeal by virtue of representing the country, tend to have a much more partisan relationship with the incumbent party.

The scale of scientific publishing continues to grow, creating overload on science journalists who are inundated with choices for what would be most interesting, important, and newsworthy to cover in their reporting. Our work addresses this problem by considering the viability of creating a predictive model of newsworthiness of scientific articles that is trained using crowdsourced evaluations of newsworthiness. We proceed by first evaluating the potential of crowd-sourced evaluations of newsworthiness by assessing their alignment with expert ratings of newsworthiness, analyzing both quantitative correlations and qualitative rating rationale to understand limitations. We then demonstrate and evaluate a predictive model trained on these crowd ratings together with arXiv article metadata, text, and other computed features. Based on the crowdsourcing protocol we developed, we find that while crowdsourced ratings of newsworthiness often align moderately with expert ratings, there are also notable differences and divergences which limit the approach. Yet despite these limitations we also find that the predictive model we built provides a reasonably precise set of rankings when validated against expert evaluations (P@10 = 0.8, P@15 = 0.67), suggesting that a viable signal can be learned from crowdsourced evaluations of newsworthiness. Based on these findings we discuss opportunities for future work to leverage crowdsourcing and predictive approaches to support journalistic work in discovering and filtering newsworthy information.