Large Language Models (LLMs) are typically evaluated through general or domain-specific benchmarks testing capabilities that often lack grounding in the lived realities of end users. Critical domains such as healthcare require evaluations that extend beyond artificial or simulated tasks to reflect the everyday needs, cultural practices, and nuanced contexts of communities. We propose Samiksha, a community-driven evaluation pipeline co-created with civil-society organizations (CSOs) and community members. Our approach enables scalable, automated benchmarking through a culturally aware, community-driven pipeline in which community feedback informs what to evaluate, how the benchmark is built, and how outputs are scored. We demonstrate this approach in the health domain in India. Our analysis highlights how current multilingual LLMs address nuanced community health queries, while also offering a scalable pathway for contextually grounded and inclusive LLM evaluation.

Numerous parties are calling for “the democratisation of AI”, but the phrase is used to refer to a variety of goals, the pursuit of which sometimes conflict. This paper identifies four kinds of “AI democratisation” that are commonly discussed: (1) the democratisation of AI use, (2) the democratisation of AI development, (3) the democratisation of AI profits, and (4) the democratisation of AI governance. Numerous goals and methods of achieving each form of democratisation are discussed. The main takeaway from this paper is that AI democratisation is a multifarious and sometimes conflicting concept that should not be conflated with improving AI accessibility. If we want to move beyond ambiguous commitments to “democratising AI”, to productive discussions of concrete policies and trade-offs, then we need to recognise the principal role of the democratisation of AI governance in navigating tradeoffs and risks across decisions around use, development, and profits.

Women’s economic empowerment is central to gender equality. However, work opportunities available to low-income women in patriarchal societies are infrequent. While crowd work has the potential to increase labor participation of such women, much remains unknown about their engagement with crowd work and the resultant opportunities and tensions. To fill this gap, we critically examined the adoption and use of a crowd work platform by low-income women in India. Through a qualitative study, we found that women faced tremendous challenges, for example, in seeking permission from family members to do crowd work, lack of family support and encouragement, and often working in unfavorable environments where they had to hide their work lives. While crowd work took a toll on their physical and emotional wellbeing, it also led to increased confidence, agency, and autonomy. We discuss ways to reduce frictions and tensions in participation of low-income women on crowd work platforms.

Social media has become increasingly important as a space for both organized and ad-hoc activism. As social movement organizations have shifted many aspects of their communication online, bystander populations - citizens who may be somewhat interested in a movement but are largely uninvolved in its activities - have begun to play a more significant role in the achievement of movement aims. In this paper, we examine bystander targeting on social media by social movement organizations in India. Through interviews, observation, and participation, we investigate ways that activists themselves conceptualize, carry out, and reflect upon their own messaging strategies in online spaces, and the perceived success and failure of these strategies. We discuss the limitations of social media as a space in which to achieve movement aims, particularly in a class-segregated context. Our work illustrates the tension faced by activists between prioritizing short and long-term movement goals in the digital sphere.

Voice-based technologies are essential to cater to the hundreds of millions of new smartphone users. However, most of the languages spoken by these new users have little to no labelled speech data. Unfortunately, collecting labelled speech data in any language is an expensive and resource-intensive task. Moreover, existing platforms typically collect speech data only from urban speakers familiar with digital technology whose dialects are often very different from low-income users. In this paper, we explore the possibility of collecting labelled speech data directly from low-income workers. In addition to providing diversity to the speech dataset, we believe this approach can also provide valuable supplemental earning opportunities to these communities. To this end, we conducted a study where we collected labelled speech data in the Marathi language from three different user groups: low-income rural users, low-income urban users, and university students. Overall, we collected 109 hours of data from 36 participants. Our results show that the data collected from low-income participants is of comparable quality to the data collected from university students (who are typically employed to do this work) and that crowdsourcing speech data from low-income rural and urban workers is a viable method of gathering speech data.

Mental illness is rapidly gaining recognition as a serious global challenge. Recent human-computer interaction (HCI) research has investigated mental health as a domain of concern, but is yet to venture into the Global South, where the problem exhibits a more complex, intersectional nature. In this paper, we review work on mental health in the Global South and present a case for HCI for Development (HCI4D) to look at mental health-both because it is an inarguably important area of concern in itself, and also because it impacts the efficacy of HCI4D interventions in other domains. We consider the role of cultural and resource-based interactions towards accessibility challenges and continuing stigma around mental health. We also identify participants' mental health as a constant consideration for HCI4D and present best practices for measuring and incorporating it. As an example, we demonstrate how both the process and the lens of aspirations-based design, a recently proposed approach for HCI4D research and design, may benefit from the consideration of mental health concerns. Our paper thus recommends a path forward for considering mental health in HCI4D, potentially leading to new research directions in addition to enriching existing ones.