Data reuse expedites scientific progress and conserves resources, yet connecting researchers with datasets available for reuse remains a challenge. The scientific community has proposed several recommendation systems to help identify relevant data and maximize reuse. However, test collections or benchmarks for evaluating the performance of dataset recommendation systems are rare, particularly for social science dataset retrieval. To address this gap, we created a novel test collection for evaluating social science dataset recommendation systems. Our collection includes 249,102 query–dataset pairs with relevance judgments, featuring 262 unique search queries and 10,749 datasets. We describe how we created this collection using datasets archived at the Inter-university Consortium for Political and Social Research (ICPSR) and the ICPSR Bibliography of Data-related Literature. Additionally, we demonstrate a potential use case by evaluating the performance of embedding-based recommendation models on our test collection. The test collection is available through ICPSR at https://doi.org/10.3886/E238682V1.

Manually annotating data for computational social science tasks can be costly, time-consuming, and emotionally draining. While recent work suggests that LLMs can perform such annotation tasks in zero-shot settings, little is known about how prompt design impacts LLMs' compliance and accuracy. We conduct a large-scale multi-prompt experiment to test how model selection (GPT-4o, GPT-3.5, PaLM2, and Falcon7b) and prompt design features (definition inclusion, output type, explanation, and prompt length) impact the compliance and accuracy of LLM-generated annotations on four highly relevant and diverse CSS tasks (toxicity, sentiment, rumor stance, and news frames). Our results show that LLM compliance and accuracy are prompt-dependent. For instance, prompting for numerical scores instead of labels reduces all LLMs' compliance and accuracy. Concise prompts can significantly reduce prompting costs but also lead to lower accuracy on tasks like toxicity. Furthermore, minor prompt changes like asking for an explanation can cause large changes in the distribution of LLM-generated labels. By assessing the impact of prompt design on the quality and distribution of LLM-generated annotations, this work serves as both a practical guide and a warning for using LLMs in CSS research.

Abstract Many theories of human information behavior (HIB) assume that information objects are in text document format. This paper argues four important HIB theories are insufficient for describing users' search strategies for data because of assumptions about the attributes of objects that users seek. We first review and compare four HIB theories: Bates' berrypicking , Marchionni's electronic information search , Dervin's sense‐making , and Meho and Tibbo's social scientist information‐seeking . All four theories assume that information‐seekers search for text documents. Next, we compare these theories to search behavior by analyzing Google Analytics data from the Inter‐university Consortium for Political and Social Research (ICPSR). Users took direct, scenic, and orienting paths when searching for data. We also interviewed ICPSR users ( n = 20), and they said they needed dataset documentation and contextual information to find data. However, Dervin's sense‐making alone cannot explain the information‐seeking behaviors that we observed. Instead, what mattered most were object attributes determined by the type of information that users sought (i.e., data, not documents). We conclude by suggesting an alternative frame for building user‐centered data discovery tools.

Abstract This study uses a theoretical lens of infrastructural dimensions to examine stakeholders' perceptions of the value of curation, focusing on the social science data repository, the Inter‐university Consortium for Political and Social Research (ICPSR). Drawing on 67 interviews with both internal (ICPSR staff) and external (funders, data producers, and reusers) stakeholders, we analyze how value is ascribed to curation across technical, organizational, and social components of infrastructure. We identify five key ways interviewees conceptualized the value of curation infrastructures: supporting sustainability and durability, enabling research efficiency, fostering trust, building community, and advancing data equity. Our findings highlight the role of curation in knowledge generation by reframing curation as infrastructure rather than a set of discrete practices. We clarify how transparency operates in dual—and sometimes conflicting—ways: as both understandability and invisibility, shaping trust in and access to data repositories. Second, we demonstrate how data equity is increasingly perceived by stakeholders as a core infrastructural value, enacted through practices that lower barriers to access. Finally, we surface the persistent challenges in evaluating and funding curation infrastructures due to their long time horizons and often‐invisible nature. This work advocates recognizing and funding curation infrastructures as essential for long‐term scientific and societal progress.

Supervised machine learning is a common approach for automated harmful content detection to support content moderation. This approach relies on data annotated by humans to train models to recognize classes of harmful content. For detection tasks, researchers or content moderation communities typically either design their own annotation tasks to generate training data for new harmful content detection models, or use off-the-shelf (OTS) pre-trained harmful content detection models. OTS model reuse can enable detection tasks in resource-constrained contexts and can help to reduce the environmental impact of training new models -- an energy-intensive process. However, given the plethora of OTS models now available for reuse, determining which OTS model to reuse for a particular task and how to use it can be challenging, especially given that many of these models have been developed for specific contexts that are not always easily transferred onto others. This work aims to provide best practices for reusing OTS models for harmful content detection tasks. By using content analysis and statistical methods to evaluate assumptions about OTS model utility and reusability, we show that model reusers cannot assume that a model claimed to detect a particular concept, will actually detect that concept. Instead, based on our findings, we offer a decision tree for how to assess whether an OTS model would be appropriate for reuse for a new harmful content detection task. This decision tree directs model reusers to critically assess concept definitions, annotation task design, and additional features specified in our content analysis codebook to identify expected model output, and consequently evaluate whether that OTS model is appropriate for reuse for a new detection task.

Generative AI has exhibited considerable potential to transform various industries and public life. The role of news media coverage of generative AI is pivotal in shaping public perceptions and judgments about this significant technological innovation. This paper provides in-depth analysis and rich insights into the temporal and spatial distribution of topics, sentiment, and substantive themes within global news coverage focusing on the latest emerging technology—generative AI. We collected a comprehensive dataset of English news articles (January 2018 to November 2023, N = 24,827) through ProQuest databases. For topic modeling, we employed the BERTopic technique and combined it with qualitative coding to identify semantic themes. Subsequently, sentiment analysis was conducted using the RoBERTa-base model. Analysis of temporal patterns in the data reveals notable variability in coverage across key topics—business, corporate technological development, regulation and security, and education—with spikes in articles coinciding with major AI developments and policy discussions. Sentiment analysis shows a predominantly neutral to positive media stance, with the business-related articles exhibiting more positive sentiment, while regulation and security articles receive a reserved, neutral to negative sentiment. Our study offers a valuable framework to investigate global news discourse and evaluate news attitudes and themes related to emerging technologies.

As content moderation becomes a central aspect of all social media platforms and online communities, interest has grown in how to make moderation decisions contestable. On social media platforms where individual communities moderate their own activities, the responsibility to address user appeals falls on volunteers from within the community. While there is a growing body of work devoted to understanding and supporting the volunteer moderators' workload, little is known about their practice of handling user appeals. Through a collaborative and iterative design process with Reddit moderators, we found that moderators spend considerable effort in investigating user ban appeals and desired to directly engage with users and retain their agency over each decision. To fulfill their needs, we designed and built AppealMod, a system that induces friction in the appeals process by asking users to provide additional information before their appeals are reviewed by human moderators. In addition to giving moderators more information, we expected the friction in the appeal process would lead to a selection effect among users, with many insincere and toxic appeals being abandoned before getting any attention from human moderators. To evaluate our system, we conducted a randomized field experiment in a Reddit community of over 29 million users that lasted for four months. As a result of the selection effect, moderators viewed only 30% of initial appeals and less than 10% of the toxically worded appeals; yet they granted roughly the same number of appeals when compared with the control group. Overall, our system is effective at reducing moderator workload and minimizing their exposure to toxic content while honoring their preference for direct engagement and agency in appeals.

Large language models (LLMs), such as OpenAI's Generative Pre-trained Transformer (GPT), are a class of language models that have demonstrated outstanding performance across a range of natural language processing (NLP) tasks. LLMs have become a highly sought-after research area because of their ability to generate human-like language and their potential to revolutionize science and technology. In this study, we conduct bibliometric and discourse analyses of scholarly literature on LLMs. Synthesizing over 5,000 publications, this article serves as a roadmap for researchers, practitioners, and policymakers to navigate the current landscape of LLMs research. We present the research trends from 2017 to early 2023, identifying patterns in research paradigms and collaborations. We start with analyzing the core algorithm developments and NLP tasks that are fundamental in LLMs research. We then investigate the applications of LLMs in various fields and domains, including medicine, engineering, social science, and humanities. Our review also reveals the dynamic, fast-paced evolution of LLMs research. Overall, this article offers valuable insights into the current state, impact, and potential of LLMs research and its applications.

Harmful textual content is pervasive on social media, poisoning online communities and negatively impacting participation. A common approach to this issue is developing detection models that rely on human annotations. However, the tasks required to build such models expose annotators to harmful and offensive content and may require significant time and cost to complete. Generative AI models have the potential to understand and detect harmful textual content. We used ChatGPT to investigate this potential and compared its performance with MTurker annotations for three frequently discussed concepts related to harmful textual content on social media: Hateful, Offensive, and Toxic (HOT). We designed five prompts to interact with ChatGPT and conducted four experiments eliciting HOT classifications. Our results show that ChatGPT can achieve an accuracy of approximately 80% when compared to MTurker annotations. Specifically, the model displays a more consistent classification for non-HOT comments than HOT comments compared to human annotations. Our findings also suggest that ChatGPT classifications align with the provided HOT definitions. However, ChatGPT classifies “hateful” and “offensive” as subsets of “toxic.” Moreover, the choice of prompts used to interact with ChatGPT impacts its performance. Based on these insights, our study provides several meaningful implications for employing ChatGPT to detect HOT content, particularly regarding the reliability and consistency of its performance, its understanding and reasoning of the HOT concept, and the impact of prompts on its performance. Overall, our study provides guidance on the potential of using generative AI models for moderating large volumes of user-generated textual content on social media.

Social scientists increasingly share data so others can evaluate, replicate, and extend their research. To understand the process of data discovery as a precursor to data use, we study prospective users’ interactions with archived data. We gathered data for 98,000 user sessions initiated at a large social science data archive, the Inter-university Consortium for Political and Social Research (ICPSR). Our data reflect four years (2012-16) of users’ interactions with archival resources, including a data catalog, study-level metadata, variables, and publications that cite nearly 10,000 datasets. We constructed a network of user interactions linking website landing (e.g., site entrances) to exit pages, from which we identified three types of paths that users take through the research data archive: direct, orienting, and scenic. We also interpreted points of failure (e.g., drop-offs) and recurring behaviors (e.g., sensemaking) that support or impede data discovery along search paths. We articulate strategies that users adopt as they navigate data search and suggest ways to enhance the accessibility of data, metadata, and the systems that organize each.

To reduce the spread of misinformation, social media platforms may take enforcement actions against offending content, such as adding informational warning labels, reducing distribution, or removing content entirely. However, both their actions and their inactions have been controversial and plagued by allegations of partisan bias. When it comes to specific content items, surprisingly little is known about what ordinary people want the platforms to do. We provide empirical evidence about a politically balanced panel of lay raters' preferences for three potential platform actions on 368 news articles. Our results confirm that on many articles there is a lack of consensus on which actions to take. We find a clear hierarchy of perceived severity of actions with a majority of raters wanting informational labels on the most articles and removal on the fewest. There was no partisan difference in terms of how many articles deserve platform actions but conservatives did prefer somewhat more action on content from liberal sources, and vice versa. We also find that judgments about two holistic properties, misleadingness and harm, could serve as an effective proxy to determine what actions would be approved by a majority of raters.

Despite large public investments in facilitating the secondary use of data, there is little information about the specific factors that predict data's reuse. Using data download logs from the Inter-university Consortium for Political and Social Research (ICPSR), this study examines how data properties, curation decisions, and repository funding models relate to data reuse. We find that datasets deposited by institutions, subject to many curatorial tasks, and whose access and preservation is funded externally, are used more often. Our findings confirm that investments in data collection, curation, and preservation are associated with more data reuse.

Data curation is the process of making a dataset fit-for-use and archivable. It is critical to data-intensive science because it makes complex data pipelines possible, studies reproducible, and data reusable. Yet the complexities of the hands-on, technical, and intellectual work of data curation is frequently overlooked or downplayed. Obscuring the work of data curation not only renders the labor and contributions of data curators invisible but also hides the impact that curators' work has on the later usability, reliability, and reproducibility of data. To better understand the work and impact of data curation, we conducted a close examination of data curation at a large social science data repository, the Inter-university Consortium for Political and Social Research (ICPSR). We asked: What does curatorial work entail at ICPSR, and what work is more or less visible to different stakeholders and in different contexts? And, how is that curatorial work coordinated across the organization? We triangulated accounts of data curation from interviews and records of curation in Jira tickets to develop a rich and detailed account of curatorial work. While we identified numerous curatorial actions performed by ICPSR curators, we also found that curators rely on a number of craft practices to perform their jobs. The reality of their work practices defies the rote sequence of events implied by many life cycle or workflow models. Further, we show that craft practices are needed to enact data curation best practices and standards. The craft that goes into data curation is often invisible to end users, but it is well recognized by ICPSR curators and their supervisors. Explicitly acknowledging and supporting data curators as craftspeople is important in creating sustainable and successful curatorial infrastructures.

This paper describes a machine learning approach for annotating and analyzing data curation work logs at ICPSR, a large social sciences data archive. The systems we studied track curation work and coordinate team decision-making at ICPSR. Archive staff use these systems to organize, prioritize, and document curation work done on datasets, making them promising resources for studying curation work and its impact on data reuse, especially in combination with data usage analytics. A key challenge, however, is classifying similar activities so that they can be measured and associated with impact metrics. This paper contributes: 1) a set of data curation activities; 2) a computational model for identifying curation actions in work log descriptions; and 3) an analysis of frequent data curation activities at ICPSR over time. We first propose a set of data curation actions to help us analyze the impact of curation work. We then use this set to annotate a set of data curation logs, which contain records of data transformations and project management decisions completed by archive staff. Finally, we train a text classifier to detect the frequency of curation actions in a large set of work logs. Our approach supports the analysis of curation work documented in work log systems as an important step toward studying the relationship between research data curation and data reuse.

Abstract Social media data (SMD) offer researchers new opportunities to leverage those data for their work in broad areas such as public opinion, digital culture, labor trends, and public health. The success of efforts to save SMD for reuse by researchers will depend on aligning data management and archiving practices with evolving norms around the capture, use, sharing, and security of datasets. This paper presents an initial foray into understanding how established practices for managing and preserving data should adapt to demands from researchers who use and reuse SMD, and from people who are subjects in SMD. We examine the data management practices of researchers who use SMD through a survey, and we analyze published articles that used data from Twitter. We discuss how researchers describe their data management practices and how these practices may differ from the management of conventional data types. We explore conceptual, technical, and ethical challenges for data archives based on the similarities and differences between SMD and other types of research data, focusing on the social sciences. Finally, we suggest areas where archives may need to revise policies, practices, and services in order to create secure, persistent, and usable collections of SMD.

Online abusive behavior affects millions and the NLP community has attempted to mitigate this problem by developing technologies to detect abuse.However, current methods have largely focused on a narrow definition of abuse to detriment of victims who seek both validation and solutions.In this position paper, we argue that the community needs to make three substantive changes: (1) expanding our scope of problems to tackle both more subtle and more serious forms of abuse, (2) developing proactive technologies that counter or inhibit abuse before it harms, and (3) reframing our effort within a framework of justice to promote healthy communities.

Academic researchers have been collecting data and pro- gramming scripts to process and analyze them for years. Re- searchers have studied the difficulty in sharing data alone, but sharing the scripts required to reproduce results has been discussed less often. At the Collective Action and So- cial Media (CASM) Lab at the Illinois Institute of Technol- ogy, we study how people use social media to engage with their communities. Our interdisciplinary team consists of students with various technical backgrounds. Since everyone in the lab needs to run code, we have developed a standard repository structure. We will share the structure definition and explain the reasoning behind our design decisions. We aim to make our data and code accessible to social scientists not trained in information retrieval, so we frame this paper from that perspective. By publicizing our approach we invite researchers with similar goals to build on our work, collab- orate on the design and implementation of modern tools to share code and data, and to suggest improvements to our process.

Twitter is increasingly becoming a medium through which constituents can lobby their elected representatives in Congress about issues that matter to them. Past research has focused on how citizens communicate with each other or how members of Congress (MOCs) use social media in general; our research examines how citizens communicate with MOCs. We contribute to existing literature through the careful examination of hundreds of citizen-authored tweets and the development of a categorization scheme to describe common strategies of lobbying on Twitter. Our findings show that contrary to past research that assumed citizens used Twitter to merely shout out their opinions on issues, citizens utilize a variety of sophisticated techniques to impact political outcomes.

As Twitter becomes a more common means for officials to communicate with their constituents, it becomes more important that we understand how officials use these communication tools. Using data from 380 members of Congress' Twitter activity during the winter of 2012, we find that officials frequently use Twitter to advertise their political positions and to provide information but rarely to request political action from their constituents or to recognize the good work of others. We highlight a number of differences in communication frequency between men and women, Senators and Representatives, Republicans and Democrats. We provide groundwork for future research examining the behavior of public officials online and testing the predictive power of officials' social media behavior.

Women and men communicate differently in both face-to-face and computer-mediated environments. We study linguistic patterns considered gendered in reviews contributed to the Internet Movie Database. IMDb has been described as a male-majority community, in which females contribute fewer reviews and enjoy less prestige than males. Analyzing reviews posted by prolific males and females, we hypothesize that females adjust their communication styles to be in sync with their male counterparts. We find evidence that while certain characteristics of "female language" persevere over time (e.g., frequent use of pronouns) others (e.g., hedging) decrease with time. Surprisingly, we also find that males often increase their use of "female" features. Our results indicate, that even when they resemble men's reviews linguistically, women's reviews still enjoy less prestige and smaller audiences.

This paper describes a community-based effort to preserve organizational knowledge and to orientate newcomers to a graduate school. It presents a very brief review of recent research on wiki use in corporate and organizational environments and initial data from two wiki implementation iterations within our academic community. We contrast use of a TWiki with that of a WetPaint wiki. Our data suggest that with low barriers to participation and a great deal of patience, wikis can be useful stores for community information and knowledge sharing.

Chapter 6 ("Georeferencing Elements in Metadata Standards") reviews the main metadata standards used for describing information objects that include geospatial references.Employing very helpful diagrams, the author explores the structure of the current main metadata standards and examines in detail their common points and differences.This comparison is then used as a departing point to open the discussion about overall interoperability across standards and the excessive complexity of the Geography Markup Language (GML) standard proposed by the Open Geospatial Consortium's.As a result, this chapter proposes a new generic structure for georeferencing and discusses the basic requirements of a geometry language that tries to meet the needs of general interoperability.The final objective of all the concepts, structures, and techniques introduced in the previous chapters is the development of systems that enable an effective retrieval of information objects related to specific geographic areas of interest.Chapter 7 ("Geographic Information Retrieval" [GIR]) presents the foundations of information retrieval based on geographic relationships.Unless GIR and text-based information retrieval can be used in combination, this chapter only explores the use of spatial characteristics for retrieval purposes.After presenting topological and geometric relations, the author introduces the main spatial operations that enable the matching of georeferenced information objects against users' spatial queries and analyses several currently running GIR systems.The final part of the chapter reviews current

The RideNow Project is designed to help individuals within a group or organization coordinate ad hoc shared rides. This paper describes three design decisions the RideNow team made in order to allow incremental adoption and evolution and to capitalize on local conditions. (1) The system allows users to interact with the system through email or Web, because we anticipate that email will be most convenient when there are few users but the Web interface will be more useful as the number of users increase. (2) The system does not force structure on user-entered data such as dates, times, and locations, instead allowing conventions to emerge. (3) We use the group's shared physical spaces to provide additional information about ride sharing activity.