Misinformation detection is a rapidly moving target, as new topics emerge and evolve in high volume on social media platforms. Annotated and fact-checked datasets are necessary for detection model training, but are laborious to curate. Thus, many misinformation detection models are trained in low-resource environments and rely on machine learning techniques to improve performance with small ground-truth datasets. Generative data augmentation methods enable topic-specific examples that increase a model's training dataset without incurring the cost and time investment associated with manual annotation. In this work, we assess the value of using generative augmentation for different classes of learning models: a classic neural model, a fine-tuned deep learning model, a reinforcement learning model, and an active learning model. We find that generated training data is not effective for all learning paradigms for the misinformation detection task, highlighting the need to use different quality measures to assess its value for low-resource machine learning tasks.

Large language models (LLMs) are becoming the standard for tackling complex natural language processing tasks, but the lack of transparency and explainability of closed-source, publicly available tools that leverage LLMs, such as chatbots, raises concern. Understanding the relationship between the training data, model parameters, and final outputs is particularly important when chatbots are engaged in dialogue from domains where information is continually evolving and the quality of available information can be poor (e.g. health, climate change, and elections). In this work, we investigate the difference between the research and reality of eliciting calibrated confidence scores from chatbots on three types of information: scientific fact, common knowledge, and misinformation. Our objective is to evaluate if a chatbot can accurately label the veracity of various information claims and express their uncertainty in natural language that humans can easily interpret. Focusing on GPT-4 and Claude-3, we find that there are considerable differences in the chatbot responses containing uncertainty and the range in confidence scores elicited. We also find that self-reported confidence scores are not well calibrated and do not directly map to accuracy of response. Our results highlight the need for developing standards for uncertainty quantification in order to help everyday users understand the quality of information they are obtaining, particularly in information gathering interactions where a chatbot can spread misinformation.

Word embeddings learn implicit biases from linguistic regularities captured by word cooccurrence statistics.By extending methods that quantify human-like biases in word embeddings, we introduce ValNorm, a novel intrinsic evaluation task and method to quantify the valence dimension of affect in human-rated word sets from social psychology.We apply Val-Norm on static word embeddings from seven languages (Chinese, English, German, Polish, Portuguese, Spanish, and Turkish) and from historical English text spanning 200 years.Val-Norm achieves consistently high accuracy in quantifying the valence of non-discriminatory, non-social group word sets.Specifically, Val-Norm achieves a Pearson correlation of ρ = 0.88 for human judgment scores of valence for 399 words collected to establish pleasantness norms in English.In contrast, we measure gender stereotypes using the same set of word embeddings and find that social biases vary across languages.Our results indicate that valence associations of non-discriminatory, non-social group words represent widely-shared associations, in seven languages and over 200 years.