Decoder-only large language models (LLMs) have been increasingly adopted to build embedding models for diverse tasks.To overcome the inherent limitations of causal attention in representation learning, many existing methods modify the attention mechanism to be bidirectional, potentially undermining LLMs' ability to extract semantic information acquired during pretraining.Meanwhile, leading unidirectional approaches often rely on extra input text to generate contextualized embeddings, inevitably increasing computational costs.In this work, we propose Causal2Vec, a general-purpose embedding model tailored to enhance the performance of decoder-only LLMs without altering their original architectures or introducing significant computational overhead.Specifically, we first employ a lightweight BERT-style model to preencode the input text into a single Contextual token, which is then prepended to the LLM's input sequence, allowing each token to capture contextualized information even without attending to future tokens.Furthermore, to mitigate the recency bias introduced by last-token pooling, we concatenate the last hidden states of Contextual and EOS tokens as the final text embedding.In practice, Causal2Vec achieves a new state-of-the-art performance on the MTEB benchmark among models trained solely on publicly available retrieval datasets.

Retrieving past similar dialogues to assist response generation in the present context is a promising approach to addressing the shortage of highly experienced professionals in various domains, such as user support and mental health counseling. However, unlike traditional retrieval, this task requires finding dialogue histories that lead to useful subsequent responses—a property we define as referenceability—rather than relying solely on superficial topic matching. Learning this property directly is difficult due to the impracticality of manually annotating large-scale data. To address this, we propose a framework that leverages large language models to generate pseudo-similar counseling dialogues with strict message-level correspondence to original sessions. We then use these generated dialogues as weak supervision for contrastive learning of an embedding model. By pairing original dialogue histories with their generated counterparts, the model learns to pull together histories that share referenceable subsequent counselor responses, even if their explicit situations differ. Experimental results show that our fine-tuned model significantly improves retrieval performance, increasing Hit@1 from 0.167 to 0.500 and MRR from 0.232 to 0.576. Human evaluations also confirm that the model retrieves more referenceable responses, improving the average score from 2.22 to 2.66.

Maximum a posteriori decoding, a commonly used method for neural machine translation (NMT), aims to maximize the estimated posterior probability.However, high estimated probability does not always lead to high translation quality.Minimum Bayes Risk (MBR) decoding (Kumar and Byrne, 2004) offers an alternative by seeking hypotheses with the highest expected utility.Inspired by Quality Estimation (QE) reranking which uses the QE model as a ranker (Fernandes et al., 2022), we propose source-based MBR (sMBR) decoding, a novel approach that utilizes quasi-sources (generated via paraphrasing or back-translation) as "support hypotheses" and a reference-free quality estimation metric as the utility function, marking the first work to solely use sources in MBR decoding.Experiments show that sMBR outperforms QE reranking and the standard MBR decoding.Our findings suggest that sMBR is a promising approach for NMT decoding.1 NMT x h 0

While large language models (LLMs) have made strides in understanding tabular data, current tabular evaluation benchmarks, such as WikiTableQuestions and WikiSQL, are focus on single-table scenarios, which cannot necessarily reflect the complexity of real-world applications. To bridge this gap, we present a \textbf{M}ulti-table and Multi-hop Question Answering (MMQA) dataset to assess LLMs' understanding and reasoning capabilities in handling multi-table tasks. The MMQA dataset demands that models perform multiple inferences by drawing evidence from various tables, which are designed to be connected with each other and require models to identify and utilize relationships such as foreign and primary keys. Then, we introduce a comprehensive evaluation framework that tailors to assess LLMs' capabilities in several aspects including Multi-Table Retrieval, Text-to-SQL Generation, Multi-Table QA, Primary Key Selection, and Foreign Key Selection. Finally, we propose a novel multi-table retrieval method that achieves state-of-the-art (SOTA) performance on the MMQA dataset compared to several strong baselines. Our experiment results reveal that, compared with human performance, both open-source and commercial LLMs leave significant performance room for improvements in multi-table understanding and reasoning tasks. We believe that the MMQA benchmark will enhance and facilitate LLMs' multi-table capabilities in real-world scenarios.

While Large Language Models (LLMs) excel in question-answering (QA) tasks, their real reasoning abilities on multiple evidence retrieval and integration on Multi-hop QA tasks remain less explored. Firstly, LLMs sometimes generate answers that rely on internal memory rather than retrieving evidence and reasoning in the given context, which brings concerns about the evaluation quality of real reasoning abilities. Although previous counterfactual QA benchmarks can separate the internal memory of LLMs, they focus solely on final QA performance, which is insufficient for reporting LLMs' real reasoning abilities. Because LLMs are expected to engage in intricate reasoning processes that involve evidence retrieval and answering a series of sub-questions from given passages. Moreover, current factual Multi-hop QA (MHQA) benchmarks are annotated on open-source corpora such as Wikipedia, although useful for multi-step reasoning evaluation, they show limitations due to the potential data contamination in LLMs' pre-training stage. To address these issues, we introduce the Step-wise and Counterfactual benchmark (CofCA), a novel evaluation benchmark consisting of factual data and counterfactual data that reveals LLMs' real reasoning abilities on multi-step reasoning and reasoning chain evaluation. Our experimental results reveal a significant performance gap of several LLMs between Wikipedia-based factual data and counterfactual data, deeming data contamination issues in existing benchmarks. Moreover, we observe that LLMs usually bypass the correct reasoning chain, showing an inflated multi-step reasoning performance. We believe that our CofCA benchmark will enhance and facilitate the evaluations of trustworthy LLMs.

The Speaker-To-Dialogue Attribution (SDA) task targets attributing the speaker for each utterance in the dialogue context from literature (novel) texts. Although previous studies have attempted to optimize model architectures and employ efficient data usage, SDA still faces the severe challenge of data deficiency due to its high annotation cost. To address this issue, following the extractive machine reading comprehension formulation (MRC) of SDA for realistic concerns, we propose the first synthetic data-driven SDA framework, which automatically synthesizes large-scale SDA training samples using drama script texts. Given the inherent differences between novel texts and scripts, we employ an MLM-Cloze synthesis approach to transform script fragments into novel-style paragraphs. Additionally, to mitigate risks introduced by synthetic data-driven training, we also propose two anti-risk strategies, namely the Joint Contrastive Learning Machine Reader framework and the Dynamic Train-Val Data Influence metric. Experiments on existing benchmarks demonstrate the effectiveness of our methodology. Our code and data are released at https://github.com/huangyiqian-h1kk/S2N.

Popularity bias occurs when popular items are recommended far more frequently than they should be, negatively impacting both user experience and recommendation accuracy. Existing debiasing methods mitigate popularity bias often uniformly across all users and only partially consider the time evolution of users or items. However, users have different levels of preference for item popularity, and this preference is evolving over time. To address these issues, we propose a novel method called CausalEPP (Causal Intervention on Evolving Personal Popularity) for taming recommendation bias, which accounts for the evolving personal popularity of users. Specifically, we first introduce a metric called Evolving Personal Popularity to quantify each user's preference for popular items. Then, we design a causal graph that integrates evolving personal popularity into the conformity effect, and apply deconfounded training to mitigate the popularity bias of the causal graph. During inference, we consider the evolution consistency between users and items to achieve a better recommendation. Empirical studies demonstrate that CausalEPP outperforms baseline methods in reducing popularity bias while improving recommendation accuracy.

Dynamic graph modeling aims to uncover evolutionary patterns in real-world systems, enabling accurate social recommendation and early detection of cancer cells. Inspired by the success of recent state space models in efficiently capturing long-term dependencies, we propose DyG-Mamba by translating dynamic graph modeling into a long-term sequence modeling problem. Specifically, inspired by Ebbinghaus' forgetting curve, we treat the irregular timespans between events as control signals, allowing DyG-Mamba to dynamically adjust the forgetting of historical information. This mechanism ensures effective usage of irregular timespans, thereby improving both model effectiveness and inductive capability. In addition, inspired by Ebbinghaus' review cycle, we redefine core parameters to ensure that DyG-Mamba selectively reviews historical information and filters out noisy inputs, further enhancing the model’s robustness. Through exhaustive experiments on 12 datasets covering dynamic link prediction and node classification tasks, we show that DyG-Mamba achieves state-of-the-art performance on most datasets, while demonstrating significantly improved computational and memory efficiency. Our code is available at https://github.com/Clearloveyuan/DyG-Mamba.

In the field of multi-document summarization (MDS), transformerbased models have demonstrated remarkable success, yet they suffer an input length limitation.Current methods apply truncation after the retrieval process to fit the context length; however, they heavily depend on manually well-crafted queries, which are impractical to create for each document set for MDS.Additionally, these methods retrieve information at a coarse granularity, leading to the inclusion of irrelevant content.To address these issues, we propose a novel retrieval-based framework that integrates query selection and document ranking and shortening into a unified process.Our approach identifies the most salient elementary discourse units (EDUs) from input documents and utilizes them as latent queries.These queries guide the document ranking by calculating relevance scores.Instead of traditional truncation, our approach filters out irrelevant EDUs to fit the context length, ensuring that only critical information is preserved for summarization.We evaluate our framework on multiple MDS datasets, demonstrating consistent improvements in ROUGE metrics while confirming its scalability and flexibility across diverse model architectures.Additionally, we validate its effectiveness through an in-depth analysis, emphasizing its ability to dynamically select appropriate queries and accurately rank documents based on their relevance scores.These results demonstrate that our framework effectively addresses context-length constraints, establishing it as a robust and reliable solution for MDS. 1 * Both authors contributed equally to this research.

Dynamic graph clustering aims to detect and track time-varying clusters in dynamic graphs, revealing the evolutionary mechanisms of complex real-world dynamic systems. Matrix factorization-based methods are promising approaches for this task; however, these methods often struggle with scalability and can be time-consuming when applied to large-scale dynamic graphs. Moreover, they tend to lack robustness and are vulnerable to real-world noisy data. To address these issues, we make three key contributions. First, to improve scalability, we propose temporal separated matrix factorization, where a single matrix is divided into multiple smaller matrices for independent factorization, resulting in faster computation. Second, to improve robustness, we introduce bi-clustering regularization, which jointly optimizes graph embedding and clustering, thereby filtering out noisy features from the graph embeddings. Third, to further enhance effectiveness and efficiency, we propose selective embedding updating, where we update only the embeddings of dynamic nodes while the embeddings of static nodes are fixed among different timestamps. Experimental results on six synthetic and five real-world benchmarks demonstrate the scalability, robustness and effectiveness of our proposed method. Source code is available at https://github.com/Clearloveyuan/DyG-MF.

Active learning seeks to achieve strong performance with fewer training samples. It does this by iteratively asking an oracle to label newly selected samples in a human-in-the-loop manner. This technique has gained increasing popularity due to its broad applicability, yet its survey papers, especially for deep active learning (DAL), remain scarce. Therefore, we conduct an advanced and comprehensive survey on DAL. We first introduce reviewed paper collection and filtering. Second, we formally define the DAL task and summarize the most influential baselines and widely used datasets. Third, we systematically provide a taxonomy of DAL methods from five perspectives, including annotation types, query strategies, deep model architectures, learning paradigms, and training processes, and objectively analyze their strengths and weaknesses. Then, we comprehensively summarize the main applications of DAL in natural language processing (NLP), computer vision (CV), data mining (DM), and so on. Finally, we discuss challenges and perspectives after a detailed analysis of current studies. This work aims to serve as a useful and quick guide for researchers in overcoming difficulties in DAL. We hope that this survey will spur further progress in this burgeoning field.

Aru Maekawa, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

The task of audio-visual event localization requires a model that can identify consistent events between audio and visual data in a video, where the two types of data may conflict or be obscured by background noise. Previous methods for audio-visual event analysis used features from all areas of frames and audio to establish relationships between them for matching. However, within different frames of the same video, most regions are similar, with variations occurring only in certain specific areas. To identify these crucial regions, we introduce a multimodal fusion approach based on graph neural networks. This involves constructing and processing graph using patchified images and audio to find the key patches essential for prediction. Our experimental results validate that our model achieves superior performance compared to previous state-of-the-art methods, even with fewer parameters on the AVE benchmark dataset.

Graph augmentation has received great attention in recent years for graph contrastive learning (GCL) to learn well-generalized node/graph representations. However, mainstream GCL methods often favor randomly disrupting graphs for augmentation, which shows limited generalization and inevitably leads to the corruption of high-level graph information, i.e., the graph community. Moreover, current knowledge-based graph augmentation methods can only focus on either topology or node features, causing the model to lack robustness against various types of noise. To address these limitations, this research investigated the role of the graph community in graph augmentation and figured out its crucial advantage for learnable graph augmentation. Based on our observations, we propose a community-invariant GCL framework to maintain graph community structure during learnable graph augmentation. By maximizing the spectral changes, this framework unifies the constraints of both topology and feature augmentation, enhancing the model’s robustness. Empirical evidence on 21 benchmark datasets demonstrates the exclusive merits of our framework. Code is released on Github (https://github.com/ShiyinTan/CI-GCL.git).

Ad text generation is vital for automatic advertising in various fields through search engine advertising (SEA) to avoid the cost problem caused by laborious human efforts for creating ad texts.Even though ad creators create the landing page (LP) for advertising and we can expect its quality, conventional approaches with reinforcement learning (RL) mostly focus on advertising keywords rather than LP information.This work investigates and shows the effective usage of LP information as a reward in RL-based ad text generation through automatic and human evaluations.Our analysis of the actually generated ad text shows that LP information can be a crucial reward by appropriately scaling its value range to improve ad text generation performance.

Continual learning (CL) with Vision-Language Models (VLMs) has overcome the constraints of traditional CL, which only focuses on previously encountered classes. During the CL of VLMs, we need not only to prevent the catastrophic forgetting on incrementally learned knowledge but also to preserve the zero-shot ability of VLMs. However, existing methods require additional reference datasets to maintain such zero-shot ability and rely on domain-identity hints to classify images across different domains. In this study, we propose Regression-based Analytic Incremental Learning (RAIL), which utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner and decouple the cross-domain correlations by projecting features to a higher-dimensional space. Cooperating with a training-free fusion module, RAIL absolutely preserves the VLM's zero-shot ability on unseen domains without any reference data. Additionally, we introduce Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting. In this setting, a CL learner is required to incrementally learn from multiple domains and classify test images from both seen and unseen domains without any domain-identity hint. We theoretically prove RAIL's absolute memorization on incrementally learned domains. Experiment results affirm RAIL's state-of-the-art performance in both X-TAIL and existing Multi-domain Task-Incremental Learning settings. The code is released at https://github.com/linghan1997/Regression-based-Analytic-Incremental-Learning.

Numerous effective methods have been developed to improve query-focused summarization (QFS) performance, e.g., pre-trained model-based and query-answer relevance-based methods. However, these methods still suffer from missing or redundant information due to the inability to capture and effectively utilize the interrelationship between the query and the source document, as well as between the source document and its generated summary, resulting in the summary being unable to answer the query or containing additional unrequired information. To mitigate this problem, we propose an end-to-end hierarchical two-stage summarization model, that first predicts essential content, and then generates a summary by emphasizing the predicted important sentences while maintaining separate encodings for the query and the source, so that it can comprehend not only the query itself but also the essential information in the source. We evaluated the proposed model on two QFS datasets, and the results indicated its overall effectiveness and that of each component.

Several methodologies have recently been proposed to enhance the performance of low-resource Neural Machine Translation (NMT). However, these techniques have yet to be explored thoroughly in the low-resource Thai and Myanmar languages. Therefore, we first applied augmentation techniques such as SwitchOut and Ciphertext Based Data Augmentation (CipherDAug) to improve NMT performance in these languages. Second, we enhanced the NMT performance by fine-tuning the pre-trained Multilingual Denoising BART model (mBART), where BART denotes Bidirectional and Auto-Regressive Transformer. We implemented three NMT systems: namely, Transformer+SwitchOut, Multi-Source Transformer+CipherDAug, and fine-tuned mBART in the bidirectional translations of Thai-English-Myanmar language pairs from the ASEAN-MT corpus. Experimental results showed that Multi-Source Transformer+CipherDAug significantly improved Bilingual Evaluation Understudy (BLEU), Character n-gram F-score (ChrF) , and Translation Error Rate (TER) scores over the first baseline Transformer and second baseline Edit-Based Transformer. The model achieved notable BLEU scores: 37.9 (English-to-Thai), 42.7 (Thai-to-English), 28.9 (English-to-Myanmar), 31.2 (Myanmar-to-English), 25.3 (Thai-to-Myanmar), and 25.5 (Myanmar-to-Thai). The fine-tuned mBART model also considerably outperformed the two baselines, except for the Myanmar-to-English pair. SwitchOut improved over the second baseline in all pairs and performed similarly to the first baseline in most cases. Last, we performed detailed analyses verifying that the CipherDAug and mBART models potentially facilitate improving low-resource NMT performance in Thai and Myanmar languages.

Speech emotion recognition (SER) has drawn increasing attention for its applications in human-machine interaction. However, existing SER methods ignore the information gap between the pre-training speech recognition task and the downstream SER task, leading to sub-optimal performance. Moreover, they require much time to fine-tune on each specific speech dataset, restricting their effectiveness in real-world scenes with large-scale noisy data. To address these issues, we propose an active learning (AL) based Fine-Tuning framework for SER that leverages task adaptation pre-training (TAPT) and AL methods to enhance performance and efficiency. Specifically, we first use TAPT to minimize the information gap between the pre-training and the downstream task. Then, AL methods are used to iteratively select a subset of the most informative and diverse samples for fine-tuning, reducing time consumption. Experiments demonstrate that using only 20% pt. samples improves 8.45% pt. accuracy and reduces 79% pt. time consumption.

With the booming development of social media, temporal link prediction (TLP), as a core technology, has been receiving increasing attention. However, current methods are based on graph neural networks, which suffer from the over-smoothing issue and easily yield indistinguishable node representations, degrading the prediction accuracy. Besides, they lack the ability to eliminate noisy temporal information and ignore the importance of high-order neighbor information for measuring the link probability between nodes. To solve these issues, we design a cross-view graph contrastive learning (GCL) framework for TLP, called Tacl. We first design two augmented views for GCL by enhancing the temporal and topological information to obtain distinguishable node representations. Then, we learn the evolution rule of temporal networks to help constrain consistency of node representations and eliminate noise. Finally, we incorporate the high-order neighbor information to measure the link probability between nodes. Extensive experiments demonstrate the effectiveness and robustness of Tacl.

Continual learning aims to accumulate knowledge to solve new tasks without catastrophic forgetting for previously learned tasks.Research on continual learning has led to the development of generative replay, which prevents catastrophic forgetting by generating pseudosamples for previous tasks and learning them together with new tasks.Inspired by the biological brain, we propose the hippocampal memory indexing to enhance the generative replay by controlling sample generation using compressed features of previous training samples.It enables the generation of a specific training sample from previous tasks, thus improving the balance and quality of generated replay samples.Experimental results indicate that our method effectively controls the sample generation and consistently outperforms the performance of current generative replay methods. 1

Multimodal emotion recognition aims to recognize emotions for each utterance of multiple modalities, which has received increasing attention for its application in human-machine interaction.Current graph-based methods fail to simultaneously depict global contextual features and local diverse uni-modal features in a dialogue.Furthermore, with the number of graph layers increasing, they easily fall into over-smoothing.In this paper, we propose a method for joint modality fusion and graph contrastive learning for multimodal emotion recognition (JOYFUL), where multimodality fusion, contrastive learning, and emotion recognition are jointly optimized.Specifically, we first design a new multimodal fusion mechanism that can provide deep interaction and fusion between the global contextual and uni-modal specific features.Then, we introduce a graph contrastive learning framework with inter-view and intra-view contrastive losses to learn more distinguishable representations for samples with different sentiments.Extensive experiments on three benchmark datasets indicate that JOYFUL achieved state-of-the-art (SOTA) performance compared to all baselines.

Multi-modal personality traits recognition aims to recognize personality traits precisely by utilizing different modality information, which has received increasing attention for its potential applications in human-computer interaction. Current methods almost fail to extract distinguishable features, remove noise, and align features from different modalities, which dramatically affects the accuracy of personality traits recognition. To deal with these issues, we propose an emotion-guided multi-modal fusion and contrastive learning framework for personality traits recognition. Specifically, we first use supervised contrastive learning to extract deeper and more distinguishable features from different modalities. After that, considering the close correlation between emotions and personalities, we use an emotion-guided multi-modal fusion mechanism to guide the feature fusion, which eliminates the noise and aligns the features from different modalities. Finally, we use an auto-fusion structure to enhance the interaction between different modalities to further extract essential features for final personality traits recognition. Extensive experiments on two benchmark datasets indicate that our method achieves state-of-the-art performance and robustness.

Text infilling aims to restore incomplete texts by filling in blanks, which has attracted more attention recently because of its wide application in ancient text restoration and text rewriting. However, attribute- aware text infilling is yet to be explored, and existing methods seldom focus on the infilling length of each blank or the number/location of blanks. In this paper, we propose an Attribute-aware Text Infilling method via a Pre-trained language model (A-TIP), which contains a text infilling component and a plug- and-play discriminator. Specifically, we first design a unified text infilling component with modified attention mechanisms and intra- and inter-blank positional encoding to better perceive the number of blanks and the infilling length for each blank. Then, we propose a plug-and-play discriminator to guide generation towards the direction of improving attribute relevance without decreasing text fluency. Finally, automatic and human evaluations on three open-source datasets indicate that A-TIP achieves state-of- the-art performance compared with all baselines.

Target-oriented Opinion Words Extraction (TOWE) is a fine-grained sentiment analysis task that aims to extract the corresponding opinion words of a given opinion target from the sentence. Recently, deep learning approaches have made remarkable progress on this task. Nevertheless, the TOWE task still suffers from the scarcity of training data due to the expensive data annotation process. Limited labeled data increase the risk of distribution shift between test data and training data. In this paper, we propose exploiting massive unlabeled data to reduce the risk by increasing the exposure of the model to varying distribution shifts. Specifically, we propose a novel Multi-Grained Consistency Regularization (MGCR) method to make use of unlabeled data and design two filters specifically for TOWE to filter noisy data at different granularity. Extensive experimental results on four TOWE benchmark datasets indicate the superiority of MGCR compared with current state-of-the-art methods. The in-depth analysis also demonstrates the effectiveness of the different-granularity filters.

Automated radiology report generation aims to generate paragraphs that describe fine-grained visual differences among cases, especially those between the normal and the diseased. Existing methods seldom consider the cross-modal alignment between textual and visual features and tend to ignore disease tags as an auxiliary for report generation. To bridge the gap between textual and visual information, in this study, we propose a “Jointly learning framework for automated disease Prediction and radiology report Generation (JPG)” to improve the quality of reports through the interaction between the main task (report generation) and two auxiliary tasks (feature alignment and disease prediction). The feature alignment and disease prediction help the model learn text-correlated visual features and record diseases as keywords so that it can output high-quality reports. Besides, the improved reports in turn provide additional harder samples for feature alignment and disease prediction to learn more precise visual and textual representations and improve prediction accuracy. All components are jointly trained in a manner that helps improve them iteratively and progressively. Experimental results demonstrate the effectiveness of JPG on the most commonly used IU X-RAY dataset, showing its superior performance over multiple state-of-the-art image captioning and medical report generation methods with regard to BLEU, METEOR, and ROUGE metrics.

Toshiki Kawamoto, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Jingyi You, Dongyuan Li, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, Hiroya Takamura. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Dynamic community detection (or graph clustering) in temporal networks has attracted much attention because it is promising for revealing the underlying mechanism of complex real-world systems. Current methods are criticized for the independence of graph representation learning and graph clustering, considerable noise during temporal information smoothing, and high time complexity. We propose a R obust T emporal S moothing C lustering method (RTSC), which involves joint graph representation learning and graph clustering, to solve these problems. RTSC can be formulated as a constrained multi-objective optimization problem. Specifically, three-order successive snapshots are first projected into the same subspace via graph embedding. We then use the embedding matrices to learn a common low-rank block-diagonal matrix that contains current clustering information and specific noise matrices with a sparse constraint to remove noise at each time step. To efficiently solve the challenging optimization problem, we also propose an optimization procedure based on the augmented Lagrangian multiplier (ALM) scheme. Experimental results on six artificial datasets and four real-world dynamic network datasets indicate that RTSC performs better than six state-of-the-art algorithms for dynamic clustering in temporal networks.

Chenlong Hu, Yukun Feng, Hidetaka Kamigaito, Hiroya Takamura, Manabu Okumura. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.

Soichiro Murakami, Sora Tanaka, Masatsugu Hangyo, Hidetaka Kamigaito, Kotaro Funakoshi, Hiroya Takamura, Manabu Okumura. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.

Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Manabu Okumura, Hiroya Takamura. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.

Sentence extractive summarization shortens a document by selecting sentences for a summary while preserving its important contents.However, constructing a coherent and informative summary is difficult using a pre-trained BERT-based encoder since it is not explicitly trained for representing the information of sentences in a document.We propose a nested tree-based extractive summarization model on RoBERTa (NeRoBERTa), where nested tree structures consist of syntactic and discourse trees in a given document.Experimental results on the CNN/DailyMail dataset showed that NeRoBERTa outperforms baseline models in ROUGE.Human evaluation results also showed that NeRoBERTa achieves significantly better scores than the baselines in terms of coherence and yields comparable scores to the state-of-the-art models.

Discourse segmentation and sentence-level discourse parsing play important roles for various NLP tasks to consider textual coherence.Despite recent achievements in both tasks, there is still room for improvement due to the scarcity of labeled data.To solve the problem, we propose a language model-based generative classifier (LMGC) for using more information from labels by treating the labels as an input while enhancing label representations by embedding descriptions for each label.Moreover, since this enables LMGC to make ready the representations for labels, unseen in the pre-training step, we can effectively use a pretrained language model in LMGC.Experimental results on the RST-DT dataset show that our LMGC achieved the state-of-the-art F 1 score of 96.72 in discourse segmentation.It further achieved the state-of-the-art relation F 1 scores of 84.69 with gold EDU boundaries and 81.18 with automatically segmented boundaries, respectively, in sentence-level discourse parsing.

Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, Masaaki Nagata. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

The recently proposed FixMatch achieved state-of-the-art results on most semi-supervised learning (SSL) benchmarks. However, like other modern SSL algorithms, FixMatch uses a pre-defined constant threshold for all classes to select unlabeled data that contribute to the training, thus failing to consider different learning status and learning difficulties of different classes. To address this issue, we propose Curriculum Pseudo Labeling (CPL), a curriculum learning approach to leverage unlabeled data according to the model's learning status. The core of CPL is to flexibly adjust thresholds for different classes at each time step to let pass informative unlabeled data and their pseudo labels. CPL does not introduce additional parameters or computations (forward or backward propagation). We apply CPL to FixMatch and call our improved algorithm FlexMatch. FlexMatch achieves state-of-the-art performance on a variety of SSL benchmarks, with especially strong performances when the labeled data are extremely limited or when the task is challenging. For example, FlexMatch achieves 13.96% and 18.96% error rate reduction over FixMatch on CIFAR-100 and STL-10 datasets respectively, when there are only 4 labels per class. CPL also significantly boosts the convergence speed, e.g., FlexMatch can use only 1/5 training time of FixMatch to achieve even better performance. Furthermore, we show that CPL can be easily adapted to other SSL algorithms and remarkably improve their performances. We open-source our code at https://github.com/TorchSSL/TorchSSL.

Sentence compression is the task of compressing a long sentence into a short one by deleting redundant words. In sequence-to-sequence (Seq2Seq) based models, the decoder unidirectionally decides to retain or delete words. Thus, it cannot usually explicitly capture the relationships between decoded words and unseen words that will be decoded in the future time steps. Therefore, to avoid generating ungrammatical sentences, the decoder sometimes drops important words in compressing sentences. To solve this problem, we propose a novel Seq2Seq model, syntactically look-ahead attention network (SLAHAN), that can generate informative summaries by explicitly tracking both dependency parent and child words during decoding and capturing important words that will be decoded in the future. The results of the automatic evaluation on the Google sentence compression dataset showed that SLAHAN achieved the best kept-token-based-F1, ROUGE-1, ROUGE-2 and ROUGE-L scores of 85.5, 79.3, 71.3 and 79.1, respectively. SLAHAN also improved the summarization performance on longer sentences. Furthermore, in the human evaluation, SLAHAN improved informativeness without losing readability.

Some downstream NLP tasks exploit discourse dependency trees converted from RST trees. To obtain better discourse dependency trees, we need to improve the accuracy of RST trees at the upper parts of the structures. Thus, we propose a novel neural top-down RST parsing method. Then, we exploit three levels of granularity in a document, paragraphs, sentences and Elementary Discourse Units (EDUs), to parse a document accurately and efficiently. The parsing is done in a top-down manner for each granularity level, by recursively splitting a larger text span into two smaller ones while predicting nuclearity and relation labels for the divided spans. The results on the RST-DT corpus show that our method achieved the state-of-the-art results, 87.0 unlabeled span score, 74.6 nuclearity labeled span score, and the comparable result with the state-of-the-art, 60.0 relation labeled span score. Furthermore, discourse dependency trees converted from our RST trees also achieved the state-of-the-art results, 64.9 unlabeled attachment score and 48.5 labeled attachment score.

We tackle the task of automatically generating a function name from source code.Existing generators face difficulties in generating low-frequency or out-of-vocabulary subwords.In this paper, we propose two strategies for copying low-frequency or out-of-vocabulary subwords in inputs.Our best performing model showed an improvement over the conventional method in terms of our modified F1 and accuracy on the Java-small and Java-large datasets.

In community-based question answering (CQA) platforms, it takes time for a user to get useful information from among many answers.Although one solution is an answer ranking method, the user still needs to read through the top-ranked answers carefully.This paper proposes a new task of selecting a diverse and non-redundant answer set rather than ranking the answers.Our method is based on determinantal point processes (DPPs), and it calculates the answer importance and similarity between answers by using BERT.We built a dataset focusing on a Japanese CQA site, and the experiments on this dataset demonstrated that the proposed method outperformed several baseline methods.

We propose neural models that can normalize text by considering the similarities of word strings and sounds.We experimentally compared a model that considers the similarities of both word strings and sounds, a model that considers only the similarity of word strings or of sounds, and a model without the similarities as a baseline.Results showed that leveraging the word string similarity succeeded in dealing with misspellings and abbreviations, and taking into account the sound similarity succeeded in dealing with phonetic substitutions and emphasized characters.So that the proposed models achieved higher F 1 scores than the baseline.

Recently, automatic trivia fact extraction has attracted much research interest.Modern search engines have begun to provide trivia facts as the information for entities because they can motivate more user engagement.In this paper, we propose a new unsupervised algorithm that automatically mines trivia facts for a given entity.Unlike previous studies, the proposed algorithm targets at a single Wikipedia article and leverages its hierarchical structure via top-down processing.Thus, the proposed algorithm offers two distinctive advantages: it does not incur high computation time, and it provides a domain-independent approach for extracting trivia facts.Experimental results demonstrate that the proposed algorithm is over 100 times faster than the existing method which considers Wikipedia categories.Human evaluation demonstrates that the proposed algorithm can mine better trivia facts regardless of the target entity domain and outperforms the existing methods.

We address the task of generating live soccer-match commentaries from play event data. This task has characteristics that (i) each commentary is only partially aligned with events, (ii) play event data contains many types of categorical and numerical attributes, (iii) live commentaries often mention player names and team names. For these reasons, we propose an encoder for play event data, which is enhanced with a gate mechanism. We also introduce an attention mechanism on events. In addition, we introduced placeholders and their reconstruction mechanism to enable the model to copy appropriate player names and team names from the input data. We conduct experiments on the play data of the English Premier League, provide a discussion on the result including generated commentaries.

Ranking comments on an online news service is a practically important task for the service provider, and thus there have been many studies on this task.However, most of them considered users' positive feedback, such as "Like"-button clicks, as a quality measure.In this paper, we address directly evaluating the quality of comments on the basis of "constructiveness," separately from user feedback.To this end, we create a new dataset including 100K+ Japanese comments with constructiveness scores (C-scores).Our experiments clarify (a) C-scores are not always related to users' positive feedback and (b) the performance of pairwise ranking models tends to be more enhanced by the variation in comments than that in articles.

We propose a global optimization method under length constraint (GOLC) for neural text summarization models. GOLC increases the probabilities of generating summaries that have high evaluation scores, ROUGE in this paper, within a desired length. We compared GOLC with two optimization methods, a maximum log-likelihood and a minimum risk training, on CNN/Daily Mail and a Japanese single document summarization data set of The Mainichi Shimbun Newspapers. The experimental results show that a state-of-the-art neural summarization model optimized with GOLC generates fewer overlength summaries while maintaining the fastest processing speed; only 6.70% overlength summaries on CNN/Daily and 7.8% on long summary of Mainichi, compared to the approximately 20% to 50% on CNN/Daily Mail and 10% to 30% on Mainichi with the other optimization methods. We also demonstrate the importance of the generation of in-length summaries for post-editing with the dataset Mainich that is created with strict length constraints. The ex- perimental results show approximately 30% to 40% improved post-editing time by use of in-length summaries.

We propose a simple and effective method to inject word-level information into characteraware neural language models.Unlike previous approaches which usually inject wordlevel information at the input of a long shortterm memory (LSTM) network, we inject it into the softmax function.The resultant model can be seen as a combination of characteraware language model and simple word-level language model.Our injection method can also be used together with previous methods.Through the experiments on 14 typologically diverse languages, we empirically show that our injection method, when used together with the previous methods, works better than the previous methods, including a gating mechanism, averaging, and concatenation of word vectors.We also provide a comprehensive comparison of these injection methods.

Naoki Kobayashi, Tsutomu Hirao, Kengo Nakamura, Hidetaka Kamigaito, Manabu Okumura, Masaaki Nagata. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Yuta Hitomi, Yuya Taguchi, Hideaki Tamori, Ko Kikuta, Jiro Nishitoba, Naoaki Okazaki, Kentaro Inui, Manabu Okumura. Proceedings of the 12th International Conference on Natural Language Generation. 2019.

Social media has become a platform where users post their messages about a wide range of topics, making it a useful source of information to summarize events such as sports matches. Previous summaries of sports matches generated using social media tended to be biased towards one of the teams, due to a high proportion of the posts used being from fans of the teams involved. This is problematic because in general people desire summaries that are free from bias and objective. To remedy this problem and generate higher quality summaries, we propose two approaches. The first is a function maximization method which measures the objectivity of each post based on its constituent words. The second is a neural network-based approach, where we use an encoder-decoder architecture. Then, we compare them with an existing approach and show promising results that indicate the effectiveness of our methods.

Small images of emojis have unique characteristics as additional information in understanding writers’ intentions. They enable social media users to emphasize their emotions and to express gestural movements in their posts. In addition to emojis, kaomojis (emoticons or facemarks) also behave in a similar way. They are composed of a sequence of characters, which are popularized especially in Asian countries. Although both emojis and kaomojis fulfill similar functions and share the same meaning that can be clues in opinion mining or sentiment analysis, the previous researches have been biased to explore emojis and kaomojis separately. In this paper, we align emojis and kaomojis together as a single token in the Japanese context to offer a bridge between them. Specifically, we aim to judge whether emojis and kaomojis share the same meaning or are similar with each other. We assume that emojis and kaomojis are both a single word in order to obtain their linguistic information with the skip-gram model. Furthermore, we present a new approach to consider the appearances of emojis and kaomojis in themselves, meaning that we explore the information of their visually similar shapes. We regard both of them as a single image to take into account their visual information with the CNN model. We merge two different perspectives toward emojis and kaomojis by exploring their linguistic and visual information simultaneously on the same space. The experimental results showed that we can align an unlimited number of emojis and kaomojis together with their representations (embeddings), and adding the visual information to the linguistic information can improve their representations.

This study proposes a new neural machine translation (NMT) model based on the encoder-decoder model that incorporates named entity (NE) tags of source-language sentences. Conventional NMT models have two problems enumerated as follows: (i) they tend to have difficulty in translating words with multiple meanings because of the high ambiguity, and (ii) these models’abilitytotranslatecompoundwordsseemschallengingbecausetheencoderreceivesaword, a part of the compound word, at each time step. To alleviate these problems, the encoder of the proposed model encodes the input word on the basis of its NE tag at each time step, which could reduce the ambiguity of the input word. Furthermore,the encoder introduces a chunk-level LSTM layer over a word-level LSTM layer and hierarchically encodes a source-language sentence to capture a compound NE as a chunk on the basis of the NE tags. We evaluate the proposed model on an English-to-Japanese translation task with the ASPEC, and English-to-Bulgarian and English-to-Romanian translation tasks with the Europarl corpus. The evaluation results show that the proposed model achieves up to 3.11 point improvement in BLEU.

Recent neural models for response generation show good results in terms of general responses.In real conversations, however, depending on the speaker/responder, similar utterances should require different responses.In this study, we attempt to consider individual user's information in adjusting the notable sequence-to-sequence (seq2seq) model for more diverse, userspecific responses.We assume that we need user-specific features to adjust the response and we argue that some selected representative words from the users are suitable for this task.Furthermore, we prove that even for unseen or unknown users, our model can provide more diverse and interesting responses, while maintaining correlation with input utterances.Experimental results with human evaluation show that our model can generate more interesting responses than the popular seq2seqmodel and achieve higher relevance with input utterances than our baseline.

We focus on non-standard usages of common words on social media.In the context of social media, words sometimes have other usages that are totally different from their original.In this study, we attempt to distinguish non-standard usages on social media from standard ones in an unsupervised manner.Our basic idea is that nonstandardness can be measured by the inconsistency between the expected meaning of the target word and the given context.For this purpose, we use context embeddings derived from word embeddings.Our experimental results show that the model leveraging the context embedding outperforms other methods and provide us with findings, for example, on how to construct context embeddings and which corpus to use.

In this research, we propose the task of question summarization. We first analyzed question-summary pairs extracted from a Community Question Answering (CQA) site, and found that a proportion of questions cannot be summarized by extractive approaches but requires abstractive approaches. We created a dataset by regarding the question-title pairs posted on the CQA site as question-summary pairs. By using the data, we trained extractive and abstractive summarization models, and compared them based on ROUGE scores and manual evaluations. Our experimental results show an abstractive method using an encoder-decoder model with a copying mechanism achieves better scores for both ROUGE-2 F-measure and the evaluations by human judges.

We present an automated, real-time tweet selection system for TV news programs. The system collects tweets related to a TV news program and chooses an appropriate tweet every 10 seconds for display. The selection procedure consists of two steps: assessing the importance of each tweet, and assessing the difference between the tweet and previously selected tweets. The previously selected tweets are taken into consideration for the purpose of maintaining the diversity of the entire set of selected tweets. We conducted experiments with a real TV news program and showed that the system developed in this study can select appropriate tweets.

The canonical word order of Japanese double object constructions has attracted considerable attention among linguists and has been a topic of many studies. However, most of these studies require either manual analyses or measurements of human characteristics such as brain activities or reading times for each example. Thus, while these analyses are reliable for the examples they focus on, they cannot be generalized to other examples. On the other hand, the trend of actual usage can be collected automatically from a large corpus. Thus, in this paper, we assume that there is a relationship between the canonical word order and the proportion of each word order in a large corpus and present a corpusbased analysis of canonical word order of Japanese double object constructions.

Neural encoder-decoder models have shown great success in many sequence generation tasks.However, previous work has not investigated situations in which we would like to control the length of encoder-decoder outputs.This capability is crucial for applications such as text summarization, in which we have to generate concise summaries with a desired length.In this paper, we propose methods for controlling the output sequence length for neural encoder-decoder models: two decoding-based methods and two learning-based methods. 1 Results show that our learning-based methods have the capability to control length without degrading summary quality in a summarization task.

In hierarchical phrase-based machine translation, a rule table is automatically learned by heuristically extracting syn-chronous rules from a parallel corpus. As a result, spuriously many rules are extracted which may be composed of various incorrect rules. The larger rule table incurs more run time for decoding and may result in lower translation quality. To resolve the problems, we propose a hierarchical back-off model for Hiero grammar, an instance of a synchronous context free grammar (SCFG), on the basis of the hierarchical Pitman-Yor process. The model can extract a compact rule and phrase table without resorting to any heuristics by hierarchically backing off to smaller phrases under SCFG. Inference is efficiently carried out using two-step synchronous parsing of Xiao et al., (2012) combined with slice sampling. In our experiments, the proposed model achieved higher or at least comparable translation quality against a previous Bayesian model on various language pairs; German/French/Spanish/Japanese-English. When compared against heuristic models, our model achieved comparable translation quality on a full size German-English language pair in Europarl v7 corpus with significantly smaller grammar size; less than 10 % of that for heuristic model. 1

The personality that is estimated based on documents of blogs or tweets in Twitter can not agree in the sender’s real personality. It is important that we recognize the difference between these estimated and real personalities. This paper constructs a knowledge-base for extracting the sender’s virtual personality in customer-generated media. We focus on sender’s emotions that are included in sender’s posts for automatic personality estimation. We examined the correlation between the ratio of each emotion term (anger, sadness, fear, disappointment, regret, guilt, shame, pleasure, and ease) in all sentences of each participant and the values of NEO-FFI (Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness to Experience) based on the experiment that human subjects who stayed in each sender’s character answered to NEO-FFI.As an evaluation result, we find out that the sender’s virtual personality is potentially-correlated with emotions in sender’s posts.

Andrew Shin, Ryohei Sasano, Hiroya Takamura, Manabu Okumura. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2015.

Generative word alignment models, such as IBM Models, are restricted to oneto-many alignment, and cannot explicitly represent many-to-many relationships in a bilingual text.The problem is partially solved either by introducing heuristics or by agreement constraints such that two directional word alignments agree with each other.In this paper, we focus on the posterior regularization framework (Ganchev et al., 2010) that can force two directional word alignment models to agree with each other during training, and propose new constraints that can take into account the difference between function words and content words.Experimental results on French-to-English and Japanese-to-English alignment tasks show statistically significant gains over the previous posterior regularization baseline.We also observed gains in Japanese-to-English translation tasks, which prove the effectiveness of our methods under grammatically different language pairs.

We present a method for automatically acquiring knowledge for case alternation between the passive and active voices in Japanese.By leveraging several linguistic constraints on alternation patterns and lexical case frames obtained from a large Web corpus, our method aligns a case frame in the passive voice to a corresponding case frame in the active voice and finds an alignment between their cases.We then apply the acquired knowledge to a case alternation task and prove its usefulness.

Twitter has emerged as a platform for crowds to express their opinions. Many Twitter users post their opinions, impressions, and statuses of televised events such as sports events. However, since the volume of such posts is extremely huge, it requires a lot of time and effort to understand what happens within events. We propose a method of generating live sports updates from Twitter posts on an event. Our method selects descriptive and prompt tweets that are posted within a short time after important sub events by exploiting users called good reporters, who promptly explain what is happening at each moment throughout the event. The experimental results indicated that our new technique generated more comprehensive updates than other methods presented in previous work.

We propose a named entity (NE) recognition method in which word chunks are repeatedly decomposed and concatenated. Our method identifies word chunks with a base chunker, such as a noun phrase chunker, and then recognizes NEs from the recognized word chunk sequences. By using word chunks, we can obtain features that cannot be obtained in word-sequence-based recognition methods, such as the first word of a word chunk, the last word of a word chunk, and so on. However, each word chunk may include a part of an NE or multiple NEs. To solve this problem, we use the following operators: SHIFT for separating the first word from a word chunk, POP for separating the last word from a word chunk, JOIN for concatenating two word chunks, and REDUCE for assigning an NE label to a word chunk. We evaluate our method on a Japanese NE recognition dataset that includes about 200,000 annotations of 191 types of NEs from over 8,500 news articles. The experimental results show that the training and processing speeds of our method are faster than those of a linear-chain structured perceptron and a semi-Markov perceptron, while maintaining high accuracy.

We propose a new model for the guided text summarization task. In this task, it is required that a generated summary covers all the aspects, which are predefined for the topic of the given document cluster; for example, aspects for the topic "Accidents and Natural Disasters" include WHAT, WHEN, WHERE, WHY, WHO AFFECTED, DAMAGES and COUNTERMEASURES. We use as a scorer for an aspect, the maximum entropy classifier that predicts whether each sentence reflects the aspect or not. We formalize the coverage of the aspects as a max-min problem, which enables a summary to cover aspects in a well-balanced manner. In the max-min problem, the minimum of the aspect scores is going to be maximized so that the summary contains all the aspects as much as possible. Furthermore, we integrate the model based on the max-min problem with the maximum coverage summarization model, which generates a summary containing as many conceptual units as possible. Through the experiments on benchmark datasets for the guided summarization, we show that our model outperforms other approaches in terms of ROUGE-2.

We propose to use a structured output learning for summary generation based on the maximum coverage problem. Our method learns a function that outputs the benefit of each conceptual unit in the document cluster for this summarization model. In the training, we iteratively run a greedy algorithm that accepts items (sentences) with different costs (length) in order to generate a summary within the given maximum length limit. We empirically show that the structured output learning works well for this task and also examine its behavior in several dierent settings.

We present a new approach to detect abbreviations given a root expression. The method is based on a statistical model combining two internal models: a generation and a verification model. The statistical model accounts for both the validity of abbrev

We propose a multi-document generic summarization model based on the budgeted median problem. Our model selects sentences to generate a summary so that every sentence in the document cluster can be assigned to and be represented by a sentence in the summary as much as possible. The advantage of this model is that it covers the entire relevant part of the document cluster through sentence assignment and can incorporate asymmetric relations between sentences such as textual entailment.

Among a huge number of blogs on the internet, only some of them are considered to have great contents and worth to be explored. We call such kind of blogs cool blogs and attempt to identify them. To solve the cool blog identification problem, we consider three assumptions on cool blogs: (1) cool blogs tend to have definite topics, (2) cool blogs tend to have sufficient amount of blog entries, and (3) cool blogs tend to have certain levels of topic consistency among their blog entries. Corresponding to these assumptions, we extract a mixture of topic probabilities using a topic model, exploit the number of blog entries of each blog, and calculate the topic consistency among blog entries using distance functions over topic probabilities, respectively. We show the benefits of the proposed assumptions through these features. A feature unification model is also presented to achieve highest effectiveness. The experimental results on Japanese blog data show that we can improve the classification results by applying proposed assumptions.

In this paper, we present a method that improves Japanese dependency parsing by using large-scale statistical information. It takes into account two kinds of information not considered in previous statistical (machine learning based) parsing methods: information about dependency relations among the case elements of a verb, and information about co-occurrence relations between a verb and its case element. This information can be collected from the results of automatic dependency parsing of large-scale corpora. The results of an experiment in which our method was used to rerank the results obtained using an existing machine learning based parsing method showed that our method can improve the accuracy of the results obtained using the existing method.

This study aims at identifying when an event written in text occurs. In particular, we classify a sentence for an event into four time-slots; morning, daytime, evening, and night. To realize our goal, we focus on expressions associated with time-slot (time-associated words). However, listing up all the time-associated words is impractical, because there are numerous time-associated expressions. We therefore use a semi-supervised learning method, the Naïve Bayes classifier backed up with the Expectation Maximization algorithm, in order to iteratively extract time-associated words while improving the classifier. We also propose to use Support Vector Machines to filter out noisy instances that indicates no specific time period. As a result of experiments, the proposed method achieved 0.864 of accuracy and outperformed other methods.

We present a system to automatically generate RSS feeds from HTML documents that consist of time-series items with date expressions, e.g., archives of weblogs, BBSs, chats, mailing lists, site update descriptions, and event announcements. Our system extracts date expressions, performs structure analysis of a HTML document, and detects or generates titles from the document.

We present a system to automatically generate RSS feeds from HTML documents that consist of time-series items with date expressions, e.g., archives of weblogs, BBSs, chats, mailing lists, site update descriptions, and event announcements. Our system extracts date expressions, performs structure analysis of a HTML document, and detects or generates titles from the document.

We propose a method for extracting semantic orientations of words: desirable or undesirable. Regarding semantic orientations as spins of electrons, we use the mean field approximation to compute the approximate probability function of the system instead of the intractable actual probability function. We also propose a criterion for parameter selection on the basis of magnetization. Given only a small number of seed words, the proposed method extracts semantic orientations with high accuracy in the experiments on English lexicon. The result is comparable to the best value ever reported.

We propose methods for extracting facemarks (emoticons) in text and classifying them into some emotional categories. In text-based communication, facemarks have gained popularity, since they help us understand what writers imply. However, there are two problems in text-based communication using facemarks; the first is the variety of facemarks and the second is lack of good comprehension in using facemarks. These problems are more serious in the areas where 2-byte characters are used, because the 2-byte characters can generate a quite large number of different facemarks. Therefore, we are going to propose methods for extraction and classification of facemarks. Regarding the extraction of facemarks as a chunking task, we automatically annotate a tag to each character in text. In the classification of the extracted facemarks, we apply the dynamic time alignment kernel (DTAK) and the string subsequence kernel (SSK) for scoring in the k-nearest neighbor (k-NN) method and for expanding usual Support Vector Machines (SVMs) to accept sequential data such as facemarks. We empirically show that our methods work well in classification and extraction of facemarks, with appropriate settings of parameters.

Recently, the prevalence of information retrieval engines has created an important application of the automatic summarization as the display of retrieval results, whereby the user can quickly and accurately judge the relevance of texts returned as a result of a query. Here, rather than producing a generic summary, the summary that reflects the user's topic of interest (information need) expressed in the query would be considered as more suitable. This type of summary is often called query‐biased summary. In this paper we present a method for producing query‐biased summaries using lexical chains. Lexical chains are sequences of words that are in lexical cohesion relation with each other, and tend to indicate fragments of a text that form a semantic unit. Using lexical chains would enable to produce more coherent and readable summaries than previous approaches to query‐biased summarization. To evaluate the effectiveness of our method, a task‐based evaluation scheme is adopted. The results from the experiments show that query‐biased summaries by lexical chains outperform others in the accuracy of subjects' relevance judgments.