Causality is fundamental in human cognition and has drawn attention in diverse research fields. With growing volumes of textual data, discerning causalities within text data is crucial, and causal text mining plays a pivotal role in extracting meaningful patterns. This study conducts comprehensive evaluations of ChatGPT’s causal text mining capabilities. Firstly, we introduce a benchmark that extends beyond general English datasets, including domain-specific and non-English datasets. We also provide an evaluation framework to ensure fair comparisons between ChatGPT and previous approaches. Finally, our analysis outlines the limitations and future challenges in employing ChatGPT for causal text mining. Specifically, our analysis reveals that ChatGPT serves as a good starting point for various datasets. However, when equipped with a sufficient amount of training data, previous models still surpass ChatGPT’s performance. Additionally, ChatGPT suffers from the tendency to falsely recognize non-causal sequences as causal sequences. These issues become even more pronounced with advanced versions of the model, such as GPT-4. In addition, we highlight the constraints of ChatGPT in handling complex causality types, including both intra/inter-sentential and implicit causality. The model also faces challenges with effectively leveraging in-context learning and domain adaptation. We release our code to support further research and development in this field.

This study focuses on discovering the ripple effects on firms of significant external events such as pandemics, resource price spikes, and natural disasters. We construct a system that presents a chain of other economic events derived from one event by extracting descriptions of causal relationships from a large amount of text data. The objective of this study is to construct a causal chain presentation system that takes into account the diversity of outputs to discover broader economic ripple effects. To accomplish this goal, we develop a new algorithm that uses Maximal Marginal Relevance to represent causal chains. Experiments using Japanese financial statement summaries confirm that this approach produces a greater variety of outputs without sacrificing accuracy. This innovative approach provides a more comprehensive understanding of how external events impact firms and their financial performance.