Alexander Miserlis Hoyle, Lorena Calvo-Bartolomé, Jordan Lee Boyd-Graber, Philip Resnik. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

A common use of NLP by social scientists is to understand large document collections. Recent data exploration and content analysis have shifted from probabilistic topic models to Large Language Models (LLMs). Yet their effectiveness in helping users understand content in real-world applications remains under explored. This study compares the knowledge users gain from unsupervised LLMs, supervised LLMs, and traditional topic models across two datasets. While unsupervised LLMs generate more human-readable topics, their topics are overly generic for domain-specific datasets and do not help users learn much about the documents. Adding human supervision to LLM generation improves data exploration by mitigating hallucination and over-genericity but requires greater human effort. Traditional topic models, such as Latent Dirichlet Allocation (LDA), remain effective for exploration but are less user-friendly. LLMs struggle to describe the haystack of large corpora without human help, particularly domain-specific data, and face scaling and hallucination limitations due to context length constraints.

Lorena Calvo-Bartolomé, Valérie Aldana, Karla Cantarero, Alonso Madroñal de Mesa, Jerónimo Arenas-García, Jordan Lee Boyd-Graber. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Topic modeling has been extensively used across disciplines to extract knowledge from text collections for policy design, implementation, and monitoring. However, flat topic models are limited in their ability to group overly broad topics with more specific ones. Hierarchical topic models ( htm ) address this limitation by offering thematic analysis at different granularity levels, but existing htm s rely on complex implementations, complicating hyperparameter tuning and domain expert-guided topic curation. This paper introduces two novel algorithms for hierarchical topic modeling: htm-ws ( htm with word selection) and htm-ds ( htm with document selection). These expert-in-the-loop methods enable domain experts to refine topic hierarchies by selecting subtopics for further partitioning. They are model-agnostic—compatible with both Bayesian and neural topic models—and require minimal hyperparameter tuning, comparable to flat topic models. Both algorithms are integrated within a tool for the training and exploitation of topic models, developed in the context of the IntelComp project. Experiments on three scientific paper datasets demonstrate the effectiveness of the algorithms through quantitative evaluations with automatic metrics and qualitative assessments via human judgment, including comparisons both between the two methods and against existing hierarchical topic models from the literature.

Analyzing document collections in Science, Technology, and Innovation (STI) is essential for informed policy-making. Advances in topic modeling offer powerful tools to uncover key themes within large, heterogeneous STI datasets that can inform decision-making. However, challenges such as aligning model outputs with expert knowledge, determining the desired topic granularity, and addressing model variability, limit their widespread adoption. This paper presents the Topic Analysis and Search Engine ( tase ), a platform developed within the European IntelComp project to address these limitations. tase combines Bayesian and Neural topic modeling techniques within a unified framework that incorporates an expert-in-the-loop approach for training and curating models. This framework is further enhanced with a Solr-based exploitation tool, featuring an innovative indexing method and a novel criterion for document retrieval. These features enable efficient semantic similarity calculations and seamless integration into decision-support systems. We demonstrate tase ’s scalability and effectiveness through two real-world STI use cases, highlighting its potential for broader applications. The software is freely available as open-source under the MIT license.