We introduce JFinTEB, the first comprehensive benchmark specifically designed for evaluating Japanese financial text embeddings. Existing embedding benchmarks provide limited coverage of language-specific and domain-specific aspects found in Japanese financial texts. Our benchmark encompasses diverse task categories including retrieval and classification tasks that reflect realistic and well-defined financial text processing scenarios. The retrieval tasks leverage instruction-following datasets and financial text generation queries, while classification tasks cover sentiment analysis, document categorization, and domain-specific classification challenges derived from economic survey data. We conduct extensive evaluations across a wide range of embedding models, including Japanese-specific models of various sizes, multilingual models, and commercial embedding services. We publicly release JFinTEB datasets and evaluation framework at https://github.com/retarfi/JFinTEB to facilitate future research and provide a standardized evaluation protocol for the Japanese financial text mining community. This work addresses a critical gap in Japanese financial text processing resources and establishes a foundation for advancing domain-specific embedding research.

In high-stakes domains such as healthcare, the reliability of Large Language Models (LLMs) is critical, particularly when generating clinical insights from incident reports. This study proposes a tag-based few-shot example selection method for prompting LLMs to generate background/causal factors and preventive measures from details of the medical incidents. For our experiments, we use the Japanese Medical Incident Dataset (JMID), a structured dataset of 3,884 real-world medical accident and near-miss reports. These reports are variably annotated with a wide range of tags--some include descriptive information (e.g., "medications," "blood transfusion therapy"). We compare three few-shot example selection strategies--random sampling, cosine similarity-based selection, and our proposed tag-based method--using GPT-4o and LLaMA 3.3. Results show that the tag-based approach achieves the highest precision and most stable generation behavior, while similarity-based selection often leads to unintended outputs and safety filter activation. These findings suggest that selecting examples based on human-interpretable dataset tags can improve generation precision and stability in clinical LLM applications.

Developing the capacity to effectively search for requisite datasets is an urgent requirement to assist data users in identifying relevant datasets considering the very limited available metadata. For this challenge, the utilization of third-party data is emerging as a valuable source for improvement. Our research introduces a new architecture for data exploration which employs a form of Retrieval-Augmented Generation (RAG) to enhance metadata-based data discovery. The system integrates large language models (LLMs) with external vector databases to identify semantic relationships among diverse types of datasets. The proposed framework offers a new method for evaluating semantic similarity among heterogeneous data sources and for improving data exploration. Our study includes experimental results on four critical tasks: 1) recommending similar datasets, 2) suggesting combinable datasets, 3) estimating tags, and 4) predicting variables. Our results demonstrate that RAG can enhance the selection of relevant datasets, particularly from different categories, when compared to conventional metadata approaches. However, performance varied across tasks and models, which confirms the significance of selecting appropriate techniques based on specific use cases. The findings suggest that this approach holds promise for addressing challenges in data exploration and discovery, although further refinement is necessary for estimation tasks.

This paper presents a knowledge management system for automobile failure analysis using retrieval-augmented generation (RAG) with large language models (LLMs) and knowledge graphs (KGs). In the automotive industry, there is a growing demand for knowledge transfer of failure analysis from experienced engineers to young engineers. However, failure events are phenomena that occur in a chain reaction, making them difficult for beginners to analyze them. While knowledge graphs, which can describe semantic relationships and structure information is effective in representing failure events, due to their capability of representing the relationships between components, there is much information in KGs, so it is challenging for young engineers to extract and understand sub-graphs from the KG. On the other hand, there is increasing interest in the use of Graph RAG, a type of RAG that combines LLMs and KGs for knowledge management. However, when using the current Graph RAG framework with an existing knowledge graph for automobile failures, several issues arise because it is difficult to generate executable queries for a knowledge graph database which is not constructed by LLMs. To address this, we focused on optimizing the Graph RAG pipeline for existing knowledge graphs. Using an original Q&A dataset, the ROUGE F1 score of the sentences generated by the proposed method showed an average improvement of 157.6% compared to the current method. This highlights the effectiveness of the proposed method for automobile failure analysis.

In this paper, we design indices of economic fluctuation narratives derived from economic surveys. Companies, governments, and investors rely on key metrics like GDP and industrial production indices to predict economic trends. However, they have yet to effectively leverage the wealth of information contained in economic text, such as causal relationships, in their economic forecasting. Therefore, we design indices of economic fluctuation from economic surveys by using our previously proposed narrative framework. From the evaluation results, it is observed that the proposed indices had a stronger correlation with cumulative lagging diffusion index than other types of diffusion indices.

We aim to enhance a price sentiment index and to more precisely understand price trends from the perspective of not only consumers but also businesses. We extract comments related to prices from the Economy Watchers Survey conducted by the Cabinet Office of Japan and classify price trends using a large language model (LLM). We classify whether the survey sample reflects the perspective of consumers or businesses, and whether the comments pertain to goods or services by utilizing information on the fields of comments and the industries of respondents included in the Economy Watchers Survey. From these classified price-related comments, we construct price sentiment indices not only for a general purpose but also for more specific objectives by combining perspectives on consumers and prices, as well as goods and services. It becomes possible to achieve a more accurate classification of price directions by employing a LLM for classification. Furthermore, integrating the outputs of multiple LLMs suggests the potential for the better performance of the classification. The use of more accurately classified comments allows for the construction of an index with a higher correlation to existing indices than previous studies. We demonstrate that the correlation of the price index for consumers, which has a larger sample size, is further enhanced by selecting comments for aggregation based on the industry of the survey respondents.

Causality is fundamental in human cognition and has drawn attention in diverse research fields. With growing volumes of textual data, discerning causalities within text data is crucial, and causal text mining plays a pivotal role in extracting meaningful patterns. This study conducts comprehensive evaluations of ChatGPT’s causal text mining capabilities. Firstly, we introduce a benchmark that extends beyond general English datasets, including domain-specific and non-English datasets. We also provide an evaluation framework to ensure fair comparisons between ChatGPT and previous approaches. Finally, our analysis outlines the limitations and future challenges in employing ChatGPT for causal text mining. Specifically, our analysis reveals that ChatGPT serves as a good starting point for various datasets. However, when equipped with a sufficient amount of training data, previous models still surpass ChatGPT’s performance. Additionally, ChatGPT suffers from the tendency to falsely recognize non-causal sequences as causal sequences. These issues become even more pronounced with advanced versions of the model, such as GPT-4. In addition, we highlight the constraints of ChatGPT in handling complex causality types, including both intra/inter-sentential and implicit causality. The model also faces challenges with effectively leveraging in-context learning and domain adaptation. We release our code to support further research and development in this field.

The domain adaptation of language models, including large language models (LLMs), has become increasingly important as the use of such models continues to expand. This study demonstrates the effectiveness of Composition to Augment Language Models (CALM) in adapting to the financial domain. CALM is a model to extend the capabilities of existing models by introducing cross-attention between two LLMs with different functions. In our experiments, we developed a CALM to enhance the financial performance of an LLM with strong response capabilities by leveraging a financial-specialized LLM. Notably, the CALM was trained using a financial dataset different from the one used to train the financial-specialized LLM, confirming CALM’s ability to adapt to various datasets. The models were evaluated through quantitative Japanese financial benchmarks and qualitative response comparisons, demonstrating that CALM enables superior responses with higher scores than the original models and baselines. Additionally, comparative experiments on connection points revealed that connecting the middle layers of the models is most effective in facilitating adaptation to the financial domain. These findings confirm that CALM is a practical approach for adapting LLMs to the financial domain.

In this study, we propose a methodology to extract, index, and visualize “climate change narratives” (stories about the connection between causal and consequential events related to climate change). We use two natural language processing methods, BERT (Bidirectional Encoder Representations from Transformers) and causal extraction, to textually analyze newspaper articles on climate change to extract “climate change narratives.” The novelty of the methodology could extract and quantify the causal relationships assumed by the newspaper’s writers. Looking at the extracted climate change narratives over time, we find that since 2018, an increasing number of narratives suggest the impact of the development of climate change policy discussion and the implementation of climate change-related policies on corporate behaviors, macroeconomics, and price dynamics. We also observed the recent emergence of narratives focusing on the linkages between climate change-related policies and monetary policy. Furthermore, there is a growing awareness of the negative impacts of natural disasters (e.g., abnormal weather and severe floods) related to climate change on economic activities, and this issue might be perceived as a new challenge for companies and governments. The methodology of this study is expected to be applied to a wide range of fields, as it can analyze causal relationships among various economic topics, including analysis of inflation expectation or monetary policy communication strategy.

Instruction tuning is essential for large language models (LLMs) to become interactive. While many instruction tuning datasets exist in English, there is a noticeable lack in other languages. Also, their effectiveness has not been well verified in non-English languages. We construct a Japanese instruction dataset by expanding and filtering existing datasets and apply the dataset to a Japanese pre-trained base model. We performed Low-Rank Adaptation (LoRA) tuning on both Japanese and English existing models using our instruction dataset. We evaluated these models from both quantitative and qualitative perspectives. As a result, the effectiveness of Japanese instruction datasets is confirmed. The results also indicate that even with relatively small LLMs, performances in downstream tasks would be improved through instruction tuning. Our instruction dataset, tuned models, and implementation are publicly available online.

The gravity models has been studied to analyze interaction between two objects such as trade amount between a pair of countries, human migration between a pair of countries and traffic flow between two cities. Particularly in the international trade, predicting trade amount is instrumental to industry and government in business decision making and determining economic policies. Whereas the gravity models well captures such interaction between objects, the model simplifies the interaction to extract essential relationships or needs handcrafted features to drive the models. Recent studies indicate the connection between graph neural networks (GNNs) and the gravity models in international trade. However, to our best knowledge, hardly any previous studies in the this domain directly predicts trade amount by GNNs. We propose GGAE (Gravity-informed Graph Auto-encoder) and its surrogate model, which is inspired by the gravity model, showing trade amount prediction by the gravity model can be formulated as an edge weight prediction problem in GNNs and solved by GGAE and its surrogate model. Furthermore, we conducted experiments to indicate GGAE with GNNs can improve trade amount prediction compared to the traditional gravity model by considering complex relationships.

This note presents a simple model of pricing data and their constituent variables traded in two-sided markets, where resale of data is allowed. The prices of those variables are exogenously set at the initial round, and in each round those prices are updated for the next round immediately at the end of the round, based on the outcomes of data transactions. If traders behave in accordance with the backward induction, then the initial prices never move in any rounds. In the subject experiment, this property was not observed but the average prices of those variables were not far from the initial values. We also examined whether information provision of the gross profits the user and non-user receive from data transactions affects the social welfare measured by the amounts of producer surplus.

We propose a new pre-trained architecture for economics/finance domain adaptation of language models in this paper. Pre-trained language models have become commonplace in a wide range of language processing applications. Because they learn from generic documents such as Wikipedia, many pre-trained language models are not fully adapted to the domain. As a result, there are two approaches: one is to develop a domain-specific pre-trained language model, and the other is to adapt the model learned on general documents to the domain through further pre-training with domain texts. However, no definitive better method has been discovered, and each project is working on it in different ways. As a result, this study focuses on the Japanese language’s economics/financial domains and investigates how pre-trained language models can be better adapted to domain-specific tasks.

This study demonstrates whether financial text is useful for tactical asset allocation using stocks by using natural language processing to create polarity indexes in financial news. In this study, we performed clustering of the created polarity indexes using the change-point detection algorithm. In addition, we constructed a stock portfolio and rebalanced it at each change point utilizing an optimization algorithm. Consequently, the asset allocation method proposed in this study outperforms the comparative approach. This result suggests that the polarity index helps construct the equity asset allocation method.

Stock embedding is a method for vector representation of stocks. There is a growing demand for vector representations of stock, i.e., stock embedding, in wealth management sectors, and the method has been applied to various tasks such as stock price prediction, portfolio optimization, and similar fund identifications. Stock embeddings have the advantage of enabling the quantification of relative relation-ships between stocks, and they can extract useful information from unstructured data such as text and network data. In this study, we propose stock embedding enhanced with textual and network information (SETN) using a domain-adaptive pre-trained transformer-based model to embed textual information and a graph neural network model to grasp network information. We evaluate the performance of our proposed model on related company information extraction tasks. We also demonstrate that stock embeddings obtained from the proposed model perform better in creating thematic funds than those obtained from baseline methods, providing a promising pathway for various applications in the wealth management industry.

In recent years, there have been growing expectations for the creation of new businesses and the improvement of the value of existing services by exchanging data in different fields. Data stored in-house within organizations have become a new source of innovation. While there is a high need for the value creation of data, determining the data value is not an easy task, as there is a wide range of factors to be considered, such as data pricing, acquisition cost, usage value, and update frequency. In this study, we observe communication, such as the sharing of know-hows in data exchange and analysis, and discuss the growing process of a community on the data platform. For the experiment, we focused on the data community in the COVID-19 disaster and used a unique dataset from the data platform Kaggle, which is the data analysis competition service. The results suggest that user actions differ in the discussion of the dataset and analysis. Moreover, providing topics, user participation, and activating actions in the early stages after the dataset is released are essential for forming a data community. We argue that the actions on the data analysis, such as comments and votes, are also crucial for fostering a common understanding of the data value.

In recent years, instead of closing data and analysis skills in-house, there has been much interest in widely releasing data analysis knowledge on the web. A data exchange platform is a type of digital platform that exchanges data between stakeholders, e.g., data owners, users, and analysts. However, the datasets handled on such platforms are independently acquired and stored by the data providers for their own purposes. These datasets are not based on the premise of coordination and combination, and there is currently little information available to discuss the systematic organization and combination of these datasets. In this study, we focus on a metadata, summary information of data, and examine the similarity of data on a data exchange platform using natural language processing. In our experiments, we use the metadata from the data exchange platform Kaggle. To compare the similarity of the data, our method employs word2vec and BERT as vectorize methods and converts data descriptions to vectors. Then, our method measures the distances of each vector by calculating cosine similarities between each vector. From experimental results, we found that Kaggle has the same character as other data exchange platforms. Additionally, the results indicated the usability of the natural language processing-based method for extracting similar data pairs.

This study demonstrates whether analysts’ sentiment toward individual stocks is useful in predicting the macroeconomic index. This can be achieved by using natural language processing to create polarity indexes from analyst reports. In this study, the created polarity indexes were analyzed using the Vector Autoregressive model with various macroeconomic indexes. Consequently, it was confirmed that the polarity indexes do have an impact on indexes such as prices, exchange rates, and government bonds.

With the development of computers and the rise of data exchange, the expectations for innovation by combining data from different industries, and data exchange platforms that handle different types of data are sprouting up. However, the data handled on such platforms have been obtained and are stored independently by data providers with different purposes, and the maintenance of data catalogs and the spread of schemata are not currently insufficient, making it difficult to understand the relationships between the data on such platforms. In this study, in order to derive the relationships among datasets and to discuss the similarity and combinability of the data on the data exchange platform, we analyze and discuss the relationships between datasets on the platform. We focussed on the data outlines and variables using the metadata of the data exchange platform service, D-Ocean, and found that the similarity of data cannot be measured by a single indicator, and that the items to be referred to calculate the similarity depending on the indicator.

Deep neural networks are powerful for text sentiment analysis; however, in the real world, they cannot be used in situations where explanations are required owing to their black-box property. In response, we propose a novel neural network model called sentiment shift neural network (SSNN) that can explain the process of its sentiment analysis prediction in a way that humans find natural and agreeable. The SSNN has the following three interpretable layers: the word-level original sentiment layer, sentiment shift layer, and word-level contextual sentiment layer. Using these layers, the SSNN can explain the process of its document-level sentiment analysis results in a human-like way. Realizing the interpretability of these layers is a crucial problem. To realize this interpretability, we propose a novel learning strategy called joint sentiment propagation (JSP) learning. Using real textual datasets, we experimentally demonstrate that the proposed JSP learning is effective for improving the interpretability of layers in SSNN and that both the predictability and explanation ability of the SSNN are high.

Abstract Although deep neural networks are excellent for text sentiment analysis, their applications in real-world practice are occasionally limited owing to their black-box property. In this study, we propose a novel neural network model called contextual sentiment neural network (CSNN) model that can explain the process of its sentiment analysis prediction in a way that humans find natural and agreeable and can catch up the summary of the contents. The CSNN has the following interpretable layers: the word-level original sentiment layer, word-level sentiment shift layer, word-level global importance layer, word-level contextual sentiment layer, and concept-level contextual sentiment layer. Because of these layers, this network can explain the process of its document-level sentiment analysis results in a human-like way using these layers. Realizing the interpretability of each layer in the CSNN is a crucial problem in the development of this CSNN because the general back-propagation method cannot realize such interpretability. To realize this interpretability, we propose a novel learning strategy called initialization propagation (IP) learning. Using real textual datasets, we experimentally demonstrate that the proposed IP learning is effective for improving the interpretability of each layer in CSNN. We then experimentally demonstrate that the CSNN has both the high predictability and high explanation ability.

Although deep neural networks are excellent for text sentiment analysis, their applications in real-world practice are occasionally limited owing to their black-box property. In response, we propose a novel neural network model called contextual sentiment neural network (CSNN) model that can explain the process of its sentiment analysis prediction in a way that humans find natural and agreeable. The CSNN has the following interpretable layers: the word-level original sentiment layer, word-level sentiment shift layer, word-level local contextual sentiment layer, word-level global importance layer, and word-level global contextual sentiment layer. Because of these layers, this network can explain the process of its document-level sentiment analysis results in a human-like way using these layers. Realizing the interpretability of each layer in the CSNN is a crucial problem in the development of this CSNN because the general back-propagation method cannot realize such interpretability. To realize this interpretability, we propose a novel learning strategy called initialization propagation (IP) learning. Using real textual datasets, we experimentally demonstrate that the proposed IP learning is effective for improving the interpretability of each layer in CSNN. We then experimentally demonstrate that both the predictability and explanation ability of the CSNN are high.