Stance classification, the task of predicting the viewpoint of an author on a subject of interest, has long been a focal point of research in domains ranging from social science to machine learning. Current stance detection methods rely predominantly on manual annotation of sentences, followed by training a supervised machine learning model. However, this manual annotation process requires laborious annotation effort, and thus hampers its potential to generalize across different contexts. In this work, we investigate the use of Large Language Models (LLMs) as a stance detection methodology that can reduce or even eliminate the need for manual annotations. We investigate 10 open source models and 7 prompting schemes, finding that LLMs are competitive with in-domain supervised models but are not necessarily consistent in their performance. We also fine-tuned the LLMs, but discovered that fine-tuning process does not necessarily lead to better performance. In general, we discover that LLMs do not routinely outperform their smaller supervised machine learning models, and thus call for stance detection to be a benchmark for which LLMs also optimize for. The code used in this study is available at https://github.com/ijcruic/LLM-Stance-Labeling .

Large Language Models (LLMs) have demonstrated remarkable capabilities in executing tasks based on natural language queries. However, these models, trained on curated datasets, inherently embody biases ranging from racial to national and gender biases. It remains uncertain whether these biases impact the performance of LLMs for certain tasks. In this study, we investigate the political biases of LLMs within the stance classification task, specifically examining whether these models exhibit a tendency to more accurately classify politically-charged stances. Utilizing three datasets, seven LLMs, and four distinct prompting schemes, we analyze the performance of LLMs on politically oriented statements and targets. Our findings reveal a statistically significant difference in the performance of LLMs across various politically oriented stance classification tasks. Furthermore, we observe that this difference primarily manifests at the dataset level, with models and prompting schemes showing statistically similar performances across different stance classification datasets. Lastly, we observe that when there is greater ambiguity in the target the statement is directed towards, LLMs have poorer stance classification accuracy.

Public opinion of military organizations plays a crucial role in their ability to recruit talented individuals. As recruitment increasingly extends into digital spaces like social media, assessing the stance of social media users toward online military content becomes essential. However, there is a notable lack of data for analyzing opinions on military recruitment efforts online, compounded by challenges in stance labeling, which is key to understanding public perceptions. Despite the importance of stance analysis for effective online military recruitment, generating human-annotated, in-domain stance labels is resource-intensive. In this paper, we address both the challenges of stance labeling and the scarcity of data on public opinions of online military recruitment by introducing and releasing the DIVERSE dataset1. This dataset comprises comments from videos on the U.S. Army’s official YouTube channel. We employed a state-of-the-art weak supervision approach, leveraging large language models to label the stance of each comment toward its respective video and the U.S. Army. Our findings indicate that the U.S. Army’s videos began attracting a substantial number of comments post-2021, with a generally balanced stance distribution among supportive, oppositional, and neutral comments, though with a slight skew towards oppositional over supportive comments.

The ability of large language models (LLMs) to perform zero-shot classification makes them viable solutions for data annotation in rapidly evolving domains where quality labeled data is often scarce and costly to obtain. However, the large-scale deployment of LLMs can be prohibitively expensive. This paper introduces an LLM chain ensemble methodology that aligns multiple LLMs in a sequence, routing data subsets to subsequent models based on classification uncertainty. This approach leverages the strengths of individual LLMs within a broader system, allowing each model to handle data points where it exhibits the highest confidence, while forwarding more complex cases to potentially more robust models. Our results show that the chain ensemble method often exceeds the performance of the best individual model in the chain and achieves substantial cost savings, making LLM chain ensembles a practical and efficient solution for large-scale data annotation challenges.

To influence the information landscape preceding and during the military invasion of Ukraine in February 2022, Russia initiated a disinformation campaign portraying Ukraine as a Nazi state. This study aims to compare discussions related to this campaign on Twitter and Telegram. The analysis reveals that the Nazis and Ukraine narrative was constant on Twitter but only emerged on Telegram after the invasion in channels that had previously focused on a broader set of conspiracy theories. Beyond the examination of Russian disinformation in this case study, the paper introduces an innovative methodology for constructing topic networks from social media data. This approach expands upon traditional topic modeling by incorporating the network properties of social media data to establish directed networks that characterize the interplay between conversation topics. Through this methodology, we gain the ability to observe topical evolutions, providing fresh insights into the disinformation campaign and its efficacy in shaping discussions around the Russian invasion on social media.