The advancement of text generation models has granted us the capability to produce coherent and convincing text on demand. Yet, in real-life circumstances, individuals do not continuously generate text or voice their opinions. For instance, consumers pen product reviews after weighing the merits and demerits of a product, and professional analysts issue reports following significant news releases. In essence, opinion expression is typically prompted by particular reasons or signals. Despite long-standing developments in opinion mining, the appropriate timing for expressing an opinion remains largely unexplored. To address this deficit, our study introduces an innovative task - the identification of news-triggered opinion expressing timing. We ground this task in the actions of professional stock analysts and develop a novel dataset for investigation. Our Generator-Assisted Decision-Focused Approach (GADFA) is decision-focused, leveraging text generation models to steer the classification model, thus enhancing overall performance. Our experimental findings demonstrate that the text generated by our model contributes fresh insights from various angles, effectively aiding in identifying the optimal timing for opinion expression.

Data-to-text generation tasks often involve processing numerical time-series as input such as financial statistics or meteorological data. Although large language models (LLMs) are a powerful approach to data-to-text, we still lack a comprehensive understanding of how well they actually understand time-series data. We therefore introduce a benchmark with 18 evaluation tasks to assess LLMs’ abilities of interpreting numerical time-series, which are categorized into: 1) event detection—identifying maxima and minima; 2) computation—averaging and summation; 3) pairwise comparison—comparing values over time; and 4) inference—imputation and forecasting. Our experiments reveal five key findings: 1) even state-of-the-art LLMs struggle with complex multi-step reasoning; 2) tasks that require extracting values or performing computations within a specified range of the time-series significantly reduce accuracy; 3) instruction tuning offers inconsistent improvements for numerical interpretation; 4) reasoning-based models outperform standard LLMs in complex numerical tasks; and 5) LLMs perform interpolation better than forecasting. These results establish a clear baseline and serve as a wake-up call for anyone aiming to blend fluent language with trustworthy numeric precision in time-series scenarios.

Opinion mining, specifically in the investment sector, has experienced a significant increase in interest over recent years. This paper presents a novel approach to overcome current limitations in assessing and ranking investor opinions based on profitability. The study introduces a pre-finetuning scheme to improve language models' capacity to distinguish professionalism, thus enabling ranking of all available opinions. Furthermore, the paper evaluates ranking results using traditional metrics and suggests the use of a pairwise setting for better performances over a regression setting. Lastly, our method is shown to be effective across various investor opinion tasks, encompassing both professional and amateur investors. The results indicate that this approach significantly enhances the efficiency and accuracy of opinion mining in the investment sector.

The task of quote attribution seeks to pair textual utterances with the name of their speakers. Despite continuing research efforts on the task, models are rarely evaluated systematically against previous models in comparable settings on the same datasets. This has resulted in a poor understanding of the relative strengths and weaknesses of various approaches. In this work we formalize the task of quote attribution, and in doing so, establish a basis of comparison across existing models. We present an exhaustive benchmark of known models, including natural extensions to larger LLM base models, on all available datasets in both English and Chinese. Our benchmarking results reveal that the CEQA model attains state-of-the-art performance among all supervised methods, and ChatGPT, operating in a four-shot setting, demonstrates performance on par with or surpassing that of supervised methods on some datasets. Detailed error analysis identify several key factors contributing to prediction errors.

We propose a method that extends a BARTbased language generator using the plug-andplay language model to control the rhetorical structure of generated text.Our approach considers rhetorical relations between clauses and generates sentences that reflect this structure using plug-and-play language models.We evaluated our method using the Newsela corpus, which consists of texts at various levels of English proficiency.Our experiments demonstrated that our method outperforms the vanilla BART in terms of the correctness of output discourse and rhetorical structures.In existing methods, the rhetorical structure tends to deteriorate when compared to the baseline, the vanilla BART, as measured by n-gram overlap metrics such as BLEU.However, our proposed method does not exhibit this significant deterioration, demonstrating its advantage.

We investigate the impact of feature respresentations derived from different language models on brain encoding models, which are designed to predict brain states from linguistic stimuli. This study aims to determine whether the variances in the feature respresentations of language models, originating from their distinct encoder/decoder architectures, training data quality and quantity, and parameter sizes, affect their predictive accuracy on brain states. By examining how these feature respresentations influence brain encoding models, we identify specific brain regions where the predictability of brain activity is consistently influenced across various models, thereby uncovering similarities in their predictive effectiveness.

We explore the idea of incorporating concepts from writing skills curricula into humanmachine collaborative writing scenarios, focusing on adding writing modes as a control for text generation models.Using crowd-sourced workers, we annotate a corpus of narrative text paragraphs with writing mode labels.Classifiers trained on this data achieve an average accuracy of ∼ 87% on held-out data.We finetune a set of large language models to condition on writing mode labels, and show that the generated text is recognized as belonging to the specified mode with high accuracy.To study the ability of writing modes to provide fine-grained control over generated text, we devise a novel turn-based text reconstruction game to evaluate the difference between the generated text and the author's intention.We show that authors prefer text suggestions made by writing mode-controlled models on average 61.1% of the time, with satisfaction scores 0.5 higher on a 5-point ordinal scale.When evaluated by humans, stories generated via collaboration with writing mode-controlled models achieve high similarity with the professionally written target story.We conclude by identifying the most common mistakes found in the generated stories.The datasets and codes are available at the Github 1 .

This study aims to clarify how functional network hubs change during hierarchical visual processing in the human brain through the estimation of brain states from features extracted using a convolutional neural network (CNN), a hierarchical model of image processing. We used representational similarity analysis for brain states predicted through encoding models based on feature representations at each layer of the CNN, and applied the PageRank algorithm to matrices converted from the generated representational dissimilarity matrices to capture the hub characteristics of brain region-related systems. This succeeded in capturing changes in the hubness of interregional brain coordination during hierarchical information processing in the human cerebral cortex in visual processing. Specifically, we found that the hubness of the occipital visual cortex increased in the early phase of visual processing, and that the hubness of the prefrontal cortex and temporal lobe increased in the late phase of visual processing. From the above, we found that our proposed method allows us to capture hierarchical changes in the hubness of interregional coordination.

In recent years, with the recent remarkable development of large-scale language models in natural language processing research, there has been an increasing number of studies employing large-scale language models to investigate the information processing mechanisms of encoding and decoding in the brain. In this study, we developed a new pre-trained language model, BrainLM, which incorporates paired data of brain activity induced by text and stimuli, and verified the accuracy of estimating brain states from natural language in multiple NLP tasks. In essence, our research has achieved several noteworthy accomplishments. Firstly, we successfully developed a multimodal model that incorporates both brain and text. Subsequently, we conducted bi-directional experiments to validate the model and ensure the reliability of both brain encoding and decoding processes. Furthermore, we performed meticulous comparative experiments, wherein we introduced 20 state-of-the-art (SOTA) language models as a control group. Our findings reveal that our proposed model outperforms superior brain encoding ability compared to the control group. Lastly, we designed a discrete Autoencoder module that extracts brain features. This module can be utilized independently to extract brain features in a wider range of brain decoding studies beyond fMRI.

Anomaly detection in multivariate time series has been attracting attention in order to realize continuous stable operation of systems. As systems diversify and monitoring targets become more complex, the number and types of measurements obtained from sensors in the system have dramatically increased. It is necessary to instantly process a large amount of complex multivariate time series in order to determine anomalies with high detection accuracy. In this paper, we proposed a Transformer with a Discriminator for Anomaly Detection in multivariate time series (TDAD). Introducing an adversarial training and attention mechanisms has improved extractions of detailed loss and time series features during model training.We compare the performance of TDAD with five other deep learning methods on five publicly available datasets and demonstrate that it can determine anomalies with high accuracy. Furthermore, by proposing a TDAD with Sparse attention mechanism (called STDAD), we improved the interpretability of the patterns of time series and achieved better results by increasing the influence of strongly relevant data points in time series with long-term dependencies.

Edison Marrese-Taylor, Yumi Hamazono, Tatsuya Ishigaki, Goran Topić, Yusuke Miyao, Ichiro Kobayashi, Hiroya Takamura. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

We propose the task of automatically generating commentaries for races in a motor racing game, from vision, structured numerical, and textual data.Commentaries provide information to support spectators in understanding events in races.Commentary generation models need to interpret the race situation and generate the correct content at the right moment.We divide the task into two subtasks: utterance timing identification and utterance generation.Because existing datasets do not have such alignments of data in multiple modalities, this setting has not been explored in depth.In this study, we introduce a new large-scale dataset that contains aligned video data, structured numerical data, and transcribed commentaries that consist of 129,226 utterances in 1,389 races in a game.Our analysis reveals that the characteristics of commentaries change depending on time and viewpoints.Our experiments on the subtasks show that it is still challenging for a state-of-the-art vision encoder to capture useful information from videos to generate accurate commentaries.We make the dataset and baseline implementation publicly available for further research.1

Lis Pereira, Xiaodong Liu, Hao Cheng, Hoifung Poon, Jianfeng Gao, Ichiro Kobayashi. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Yui Uehara, Tatsuya Ishigaki, Kasumi Aoki, Hiroshi Noji, Keiichi Goshima, Ichiro Kobayashi, Hiroya Takamura, Yusuke Miyao. Proceedings of the 28th International Conference on Computational Linguistics. 2020.

End-to-end models on data-to-text learn the mapping of data and text from the aligned pairs in the dataset.However, these alignments are not always obtained reliably, especially for the time-series data, for which real time comments are given to some situation and there might be a delay in the comment delivery time compared to the actual event time.To handle this issue of possible noisy alignments in the dataset, we propose a neural network model with multitimestep data and a copy mechanism, which allows the models to learn the correspondences between data and text from the dataset with noisier alignments.We focus on generating market comments in Japanese that are delivered each time an event occurs in the market.The core idea of our approach is to utilize multitimestep data, which is not only the latest market price data when the comment is delivered, but also the data obtained at several timesteps earlier.On top of this, we employ a copy mechanism that is suitable for referring to the content of data records in the market price data.We confirm the superiority of our proposal by two evaluation metrics and show the accuracy improvement of the sentence generation using the time series data by our proposed method.

Hayate Iso, Yui Uehara, Tatsuya Ishigaki, Hiroshi Noji, Eiji Aramaki, Ichiro Kobayashi, Yusuke Miyao, Naoaki Okazaki, Hiroya Takamura. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.

Kasumi Aoki, Akira Miyazawa, Tatsuya Ishigaki, Tatsuya Aoki, Hiroshi Noji, Keiichi Goshima, Ichiro Kobayashi, Hiroya Takamura, Yusuke Miyao. Proceedings of the 12th International Conference on Natural Language Generation. 2019.

Humans perceive continuous high-dimensional information by dividing it into significant segments such as words and units of motion. We believe that such unsupervised segmentation is also important for robots to learn topics such as language and motion. To this end, we previously proposed a hierarchical Dirichlet process-Gaussian process-hidden semi-Markov model (HDP-GP-HSMM). However, an important drawback to this model is that it cannot divide high-dimensional time-series data. Further, low-dimensional features must be extracted in advance. Segmentation largely depends on the design of features, and it is difficult to design effective features, especially in the case of high-dimensional data. To overcome this problem, this paper proposes a hierarchical Dirichlet process-variational autoencoder-Gaussian process-hidden semi-Markov model (HVGH). The parameters of the proposed HVGH are estimated through a mutual learning loop of the variational autoencoder and our previously proposed HDP-GP-HSMM. Hence, HVGH can extract features from high-dimensional time-series data, while simultaneously dividing it into segments in an unsupervised manner. In an experiment, we used various motion-capture data to show that our proposed model estimates the correct number of classes and more accurate segments than baseline methods. Moreover, we show that the proposed method can learn latent space suitable for segmentation.

Tatsuya Aoki, Akira Miyazawa, Tatsuya Ishigaki, Keiichi Goshima, Kasumi Aoki, Ichiro Kobayashi, Hiroya Takamura, Yusuke Miyao. Proceedings of the 11th International Conference on Natural Language Generation. 2018.

Humans recognize perceived continuous information by dividing it into significant segments such as words and unit motions. We believe that such unsupervised segmentation is also an important ability that robots need to learn topics such as language and motions. Hence, in this paper, we propose a method for dividing continuous time-series data into segments in an unsupervised manner. To this end, we proposed a method based on a hidden semi-Markov model with Gaussian process (GP-HSMM). If Gaussian processes, which are nonparametric models, are used, unit motion patterns can be extracted from complicated continuous motion. However, this approach requires the number of classes of segments in the time-series data in advance. To overcome this problem, in this paper, we extend GP-HSMM to a nonparametric Bayesian model by introducing a hierarchical Dirichlet process (HDP) and propose the hierarchical Dirichlet processes-Gaussian process-hidden semi-Markov model (HDP-GP-HSMM). In the nonparametric Bayesian model, an infinite number of classes is assumed and it becomes difficult to estimate the parameters naively. Instead, the parameters of the proposed HDP-GP-HSMM are estimated by applying slice sampling. In the experiments, we use various synthetic and motion-capture data to show that our proposed model can estimate a more correct number of classes and achieve more accurate segmentation than baseline methods.

Quantitative modeling of human brain activity based on language representations has been actively studied in systems neuroscience. However, previous studies examined word-level representation, and little is known about whether we could recover structured sentences from brain activity. This study attempts to generate natural language descriptions of semantic contents from human brain activity evoked by visual stimuli. To effectively use a small amount of available brain activity data, our proposed method employs a pre-trained image-captioning network model using a deep learning framework. To apply brain activity to the image-captioning network, we train regression models that learn the relationship between brain activity and deep-layer image features. The results demonstrate that the proposed model can decode brain activity and generate descriptions using natural language sentences. We also conducted several experiments with data from different subsets of brain regions known to process visual stimuli. The results suggest that semantic information for sentence generations is widespread across the entire cortex.

In this study, we investigate whether sparse coding helps explain the semantic representation in human cerebral cortex. We show this by using sparse coding to model semantic representation in the cerebral cortex. We propose three methods for estimating semantic representation from brain activity data. For estimating a new semantic representation, in the first method, we use only a semantic representation dictionary obtained via sparse coding. The semantic representation estimated using this method is more similar to the actual semantic representation of the cerebral cortex than that estimated without sparse coding. In the second method, we use only a brain activity dictionary obtained via sparse coding. The semantic representation estimated using this method is also better than that estimated without sparse coding. In addition, in the third method, we estimate semantic representation by applying sparse coding to both semantic representation and brain activity data. The semantic representation estimated by using this third method is better than that estimated by the first or second methods. Through the above three experiments, we have confirmed that sparse coding helps explain the semantic representation in human cerebral cortex.

There are quite a few high dimensional time-series data co-ocurring each other such as lip motions, voices, and face appearances and so on. When capturing the correspondent relationships among those time-series data with different dimensionality, we need to make the dimensionality all the same size so that they can be compared each other. To achieve this, Gaussian Process Latent Variable Models (GPLVM) is often used to reduce the size of high dimensional time-series data. In this study, we propose a method to introduce MLP to GPLVM-based methods in estimating latent states. We applied the proposed method to GPLVM, SharedGPLVM, GPDM, and SharedGPDM, and then confirmed that our method outperforms the conventional methods in terms of efficiency and precisely estimation.

This paper describes a method to verbalize the trends of time-series data. As an example of time-series data, we use the price of Nikkei stock average and develop a method to generate natural language sentences which describe how the stock price goes in the market. As the basic idea for making linguistic descriptions of the stock price trends, we firstly classify all the time-series data including a newly observed time-series data, i.e., the target to be verbalized, by means of spectral clustering employing Dynamic Time Warping distance as its similarity metric. Secondly, a bi-gram language model for the newly observed data is built based on the weighted bi-gram language models of the other time-series data classified in the same cluster. The weights for the bi-gram model of the target data from other time-series data are decided based on the similarity between the target data and the other data in the same cluster. Lastly, linguistic summarization for the target data is generated by finding the most likely combination of words by means of dynamic programming, employing the weighted bi-gram model. Through the experiments under the conditions of various cluster numbers in spectral clustering, we have confirmed that natural language sentences, which properly describe the trends of the stock price, are generated by our method.

Muhammad Attamimi, Yuji Ando, Tomoaki Nakamura, Takayuki Nagai, Daichi Mochihashi, Ichiro Kobayashi, Hideki Asoh. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015.

In the case that a robot controller is trained by means of evolutionary computation, the robot will be able to behave sufficiently in the environment where the robot has been trained. However, if the robot is put in an environment which is more complex than a training environment, it cannot behave sufficiently and is required to be trained again so as it fits to the complex environment. Based on this fact, we build a training environment for a robot controller with the partial components of a more complex environment than the training environment and aim to obtain a controller which makes a robot be able to act in the complex environment by only training the controller at a simpler environment. We clarify a way of building a training environment which functions effectively for training a robot controller and discuss how much training is necessary in the training environment for a robot to be able to behave under a more complex environment.

This paper proposes a general framework for a system with a semantic browsing and visualization interface called Knowledge Communication, Collaboration and Creation Browser (KC3 Browser) which integrates multimedia contests and web services on the grid networks, and makes a semantic mash-up called knowledge workspace (k-workspace) with various visual gadgets according to user's contexts (e.g. their interests, purpose and computational environments). KC3 Browser also achieves a link-free browsing for seamless knowledge access by generating semantic links based on an arbitrary knowledge models such as ontology and vector space models. It assists users to look down and to figure out various social and natural events from the web contents. We have implemented a prototype of KC3 Browser and tested it to an international project on risk intelligence against natural disaster.

Recently as digital cameras and web cameras have been commonly used in our everyday lives, we have become able to easily obtain quite a few movies. However, there are some problems in terms of managing movies, for example, it is difficult to find particular scenes in a movie, etc. As a first step toward finding particular scenes, we focus on retrieving scenes where human activity is recorded and annotating linguistic labels on the scenes. In order to annotate the labels, which are used as clues to retrieve, on the scenes, we propose a method to verbalize human behaviors in a movie with image recognition technology. Based on our proposed method, we aim to develop a method that enables us to retrieve particular human behaviors in a movie by words.

As we know that human intellectual activities are performed by the use of language; a human being thinks and reasons with language. In this paper, we propose a new simulation method which imitates human information processing based on the use of language. — An ordinary simulation method uses mathematical equations to estimate the future trends of a dynamic system, however, we propose a new simulation method which uses a model consists of natural language, which is called a linguistic model. A linguistic model does not have a fixed structure, but the structure of the model itself will be generated so that model should become the most appropriate model to estimate the future behavior of a dynamic system. We regard that human intellectual activities such as thinking, reasoning, etc. are also texts realized by context as well as a linguistic text is done by context. We show a human linguistic reasoning model generation with an example of forecast of foreign exchange rate (FER) changes.

The aim of this study is to develop a new communication protocol which has the same characteristics of natural language in terms of providing flexible communication. In this paper, we describe how natural language-based protocol is developed as a computer communication protocol and the mechanism of the communication.

This paper discusses an application of the idea of everyday language computing (Kobayashi and Sugeno, Proc. 12th Fuzzy Syst. Symp., 1996, pp. 63–66; Kobayashi and Sugeno, Proc. Breakthrough Opportunities for Fuzzy Logic, 1996, pp. 89–94), and explains that the organization of human intelligence takes the same form as that of the linguistic system. We introduce fuzziness into describing knowledge so that it becomes flexible and appropriate for the changing situations. As an example, we apply our method to the forecast of atmospheric pressure distribution (FAPD). © 1998 John Wiley & Sons, Inc.

We focus on human information processing characterized by information fusion, and propose a new simulation method for a social system. We consider the role of custom and language in human intelligence, and apply it to information processing. We use natural language as a tool for information fusion by a computer. In this context we discuss a simulation model imitating the human thinking process. As an example, we build a model to estimate the future trends of foreign exchange rates.