Annotators' sociodemographic backgrounds (i.e., the individual compositions of their gender, age, educational background, etc.) have a strong impact on their decisions when working on subjective NLP tasks, such as toxic language detection.Often, heterogeneous backgrounds result in high disagreements.To model this variation, recent work has explored sociodemographic prompting, a technique, which steers the output of prompt-based models towards answers that humans with specific sociodemographic profiles would give.However, the available NLP literature disagrees on the efficacy of this technique -it remains unclear for which tasks and scenarios it can help, and the role of the individual factors in sociodemographic prompting is still unexplored.We address this research gap by presenting the largest and most comprehensive study of sociodemographic prompting today.We analyze its influence on model sensitivity, performance and robustness across seven datasets and six instruction-tuned model families.We show that sociodemographic information affects model predictions and can be beneficial for improving zero-shot learning in subjective NLP tasks.However, its outcomes largely vary for different model types, sizes, and datasets, and are subject to large variance with regards to prompt formulations.Most importantly, our results show that sociodemographic prompting should be used with care for sensitive applications, such as toxicity annotation or when studying LLM alignment.1

Fajri Koto, Tilman Beck, Zeerak Talat, Iryna Gurevych, Timothy Baldwin. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Tilman Beck, Ji-Ung Lee, Christina Viehmann, Marcus Maurer, Oliver Quiring, Iryna Gurevych. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Transformer models are expensive to fine-tune, slow for inference, and have large storage requirements.Recent approaches tackle these shortcomings by training smaller models, dynamically reducing the model size, and by training light-weight adapters.In this paper, we propose AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions.We show that Adap-terDrop can dynamically reduce the computational overhead when performing inference over multiple tasks simultaneously, with minimal decrease in task performances.We further prune adapters from AdapterFusion, which improves the inference efficiency while maintaining the task performances entirely.

We experiment with two recent contextualized word embedding methods (ELMo and BERT) in the context of open-domain argument search.For the first time, we show how to leverage the power of contextualized word embeddings to classify and cluster topic-dependent arguments, achieving impressive results on both tasks and across multiple datasets.For argument classification, we improve the state-of-the-art for the UKP Sentential Argument Mining Corpus by 20.8 percentage points and for the IBM Debater -Evidence Sentences dataset by 7.4 percentage points.For the understudied task of argument clustering, we propose a pre-training step which improves by 7.8 percentage points over strong baselines on a novel dataset, and by 12.3 percentage points for the Argument Facet Similarity (AFS) Corpus. 1