Media bias is a multifaceted problem, leading to one-sided views and impacting decision-making. A way to address digital media bias is to detect and indicate it automatically through machine-learning methods. However, such detection is limited due to the difficulty of obtaining reliable training data. Human-in-the-loop-based feedback mechanisms have proven an effective way to facilitate the data-gathering process. Therefore, we introduce and test feedback mechanisms for the media bias domain, which we then implement on NewsUnfold, a news-reading web application to collect reader feedback on machine-generated bias highlights within online news articles. Our approach augments dataset quality by significantly increasing inter-annotator agreement by 26.31% and improving classifier performance by 2.49%. As the first human-in-the-loop application for media bias, the feedback mechanism shows that a user-centric approach to media bias data collection can return reliable data while being scalable and evaluated as easy to use. NewsUnfold demonstrates that feedback mechanisms are a promising strategy to reduce data collection expenses and continuously update datasets to changes in context.

Defining complex, evolving concepts in academic research and extracting clear taxonomies from many publications is challenging. To streamline systematic reviews and capture shifts in conceptual understanding, we present our ongoing work on TaxoMatic - a framework leveraging Large Language Models (LLMs) to automate definition extraction from academic literature. The framework encompasses data collection, relevance classification to identify papers with definitions, and definition extraction using LLMs. As a first case study, we tested our relevancy evaluation component on 2,398 articles on media bias, a domain particularly rich in varying definitions and sub-concepts. Then, we evaluated our definition extraction component on manually reviewed papers, yielding 123 definitions from 113 relevant articles. Among five tested LLMs, Claude-3-sonnet achieved the highest F1 score (0.381) for relevance classification and demonstrated a median cosine similarity of 0.557 for definition extraction with role prompting. Future directions include improving relevance classification, expanding ground truth datasets, and applying this framework to other domains, potentially enhancing conceptual clarity across disciplines.

Marking biased texts effectively increases media bias awareness, but its sustainability across new topics and unmarked news remains unclear, and the role of AI-generated bias labels is untested. This study examines how news consumers learn to perceive media bias from human- and AI-generated labels and identify biased language through highlighting, neutral rephrasing, and political orientation cues. We conducted two experiments with a teaching phase exposing them to various bias-labeling conditions and a testing phase evaluating their ability to classify biased sentences and detect biased text in unlabeled news on new topics. We find that, compared to the control group, both human- and AI-generated sentential bias labels significantly improve bias classification ( p < .001), though human labels are more effective ( d = 0.42 vs. d = 0.23). Additionally, among all teaching interventions, participants best detect biased sentences when taught with biased sentence or phrase labels ( p < .001), while politicized phrase labels reduce accuracy. The effectiveness of different media literacy interventions remains independent of political ideology, but conservative participants are generally less accurate ( p = .011), suggesting an interaction between political inclinations and bias detection. Our research provides a novel experimental framework into assessing the generalizability of media bias awareness and offer practical implications for designing bias indicators in news-reading platforms and media literacy curricula.