Environmental experts have developed the DPSIR (Driver, Pressure, State, Impact, Response) framework to systematically study and communicate key relationships between society and the environment. Using this framework requires experts to construct a DPSIR taxonomy from a corpus, annotate the documents, and identify DPSIR variables and relationships, which is laborious and inflexible. Automating it with conventional text mining faces technical challenges, primarily because the taxonomy often begins with abstract definitions, which experts progressively refine and contextualize as they annotate the corpus. In response, we develop GreenMine, a system that supports interactive text mining with prompt engineering. The system implements a prompting pipeline consisting of three simple and evaluable subtasks. In each subtask, the DPSIR taxonomy can be defined in natural language and iteratively refined as experts analyze the corpus. To support users evaluate the taxonomy, we introduce an uncertainty score based on response consistency. Then, we design a radial uncertainty chart that visualizes uncertainties and corpus topics, which supports interleaved evaluation and exploration. Using the system, experts can progressively construct the DPSIR taxonomy and annotate the corpus with LLMs. Using real-world interview transcripts, we present a case study to demonstrate the capability of the system in supporting interactive mining of DPSIR relationships, and an expert review in the form of collaborative discussion to understand the potential and limitations of the system. We discuss the lessons learned from developing the system and future opportunities for supporting interactive text mining in knowledge-intensive tasks for other application scenarios.

Recent advancements in Large Language Models (LLMs) and Prompt Engineering have made chatbot customization more accessible, significantly reducing barriers to tasks that previously required programming skills. However, prompt evaluation, especially at the dataset scale, remains complex due to the need to assess prompts across thousands of test instances within a dataset. Our study, based on a comprehensive literature review and pilot study, summarized five critical challenges in prompt evaluation. In response, we introduce a feature-oriented workflow for systematic prompt evaluation. In the context of text summarization, our workflow advocates evaluation with summary characteristics (feature metrics) such as complexity, formality, or naturalness, instead of using traditional quality metrics like ROUGE. This design choice enables a more user-friendly evaluation of prompts, as it guides users in sorting through the ambiguity inherent in natural language. To support this workflow, we introduce Awesum, a visual analytics system that facilitates identifying optimal prompt refinements for text summarization through interactive visualizations, featuring a novel Prompt Comparator design that employs a BubbleSet-inspired design enhanced by dimensionality reduction techniques. We evaluate the effectiveness and general applicability of the system with practitioners from various domains and found that (1) our design helps overcome the learning curve for non-technical people to conduct a systematic evaluation of summarization prompts, and (2) our feature-oriented workflow has the potential to generalize to other NLG and image-generation tasks. For future works, we advocate moving towards feature-oriented evaluation of LLM prompts and discuss unsolved challenges in terms of human-agent interaction.

Sensemaking on a large collection of documents (corpus) is a challenging task often found in fields such as market research, legal studies, intelligence analysis, political science, or computational linguistics. Previous works approach this problem from topic- and entity-based perspectives, but the capability of the underlying NLP model limits their effectiveness. Recent advances in prompting with LLMs present opportunities to enhance such approaches with higher accuracy and customizability. However, poorly designed prompts and visualizations could mislead users into falsely interpreting the visualizations and hinder the system's trustworthiness. In this paper, we address this issue by taking into account the user analysis tasks and visualization goals in the prompt-based data extraction stage, thereby extending the concept of Model Alignment. We present HINTs, a VA system for supporting sensemaking on large collections of documents, combining previous entity-based and topic-based approaches. The visualization pipeline of HINTs consists of three stages. First, entities and topics are extracted from the corpus with prompts. Then, the result is modeled as a hypergraph and hierarchically clustered. Finally, an enhanced space-filling curve layout is applied to visualize the hypergraph for interactive exploration. The system further integrates an LLM-based intelligent chatbot agent in the interface to facilitate the sensemaking of interested documents. To demonstrate the generalizability and effectiveness of the HINTs system, we present two case studies on different domains and a comparative user study. We report our insights on the behavior patterns and challenges when intelligent agents are used to facilitate sensemaking. We find that while intelligent agents can address many challenges in sensemaking, the visual hints that visualizations provide are still necessary. We discuss limitations and future work for combining interactive visualization and LLMs more profoundly to better support corpus analysis.