Sentence simplification, which rewrites a sentence to be easier to read and understand, is a promising technique to help people with various reading difficulties. With the rise of advanced large language models (LLMs), evaluating their performance in sentence simplification has become imperative. Recent studies have used both automatic metrics and human evaluations to assess the simplification abilities of LLMs. However, the suitability of existing evaluation methodologies for LLMs remains in question. First, the suitability of current automatic metrics on LLMs’ simplification evaluation is still uncertain. Second, current human evaluation approaches in sentence simplification often fall into two extremes: they are either too superficial, failing to offer a clear understanding of the models’ performance, or overly detailed, making the annotation process complex and prone to inconsistency, which in turn affects the evaluation’s reliability. To address these problems, this study provides in-depth insights into LLMs’ performance while ensuring the reliability of the evaluation. We design an error-based human annotation framework to assess the LLMs’ simplification capabilities. We select both closed source and open source LLMs, including GPT-4, Qwen2.5-72B, and Llama-3.2-3B. We believe that these models offer a representative selection across large, medium, and small sizes of LLMs (our corpus is available at https://github.com/WuXuanxin/human-eval-llm-simplification ). Results show that LLMs generally generate fewer erroneous simplification outputs compared to the previous state-of-the-art. However, LLMs have their limitations, as seen in GPT-4’s and Qwen2.5-72B’s struggle with lexical paraphrasing. Furthermore, we conduct meta-evaluations on widely used automatic metrics using our human annotations. We find that these metrics lack sufficient sensitivity to assess the overall high-quality simplifications, particularly those generated by high-performance LLMs .

Guanlin Li, Yuki Arase, Noel Crespi. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

We developed a controllable paraphrase generation model for semantic and lexical similarities using a simple and intuitive mechanism: attaching tags to specify these values at the head of the input sentence. Lexically diverse paraphrases have been long coveted for data augmentation. However, their generation is not straightforward because diversifying surfaces easily degrades semantic similarity. Furthermore, our experiments revealed two critical features in data augmentation by paraphrasing: appropriate similarities of paraphrases are highly downstream task-dependent, and mixing paraphrases of various similarities negatively affects the downstream tasks. These features indicated that the controllability in paraphrase generation is crucial for successful data augmentation. We tackled these challenges by fine-tuning a pre-trained sequence-to-sequence model employing tags that indicate the semantic and lexical similarities of synthetic paraphrases selected carefully based on the similarities. The resultant model could paraphrase an input sentence according to the tags specified. Extensive experiments on data augmentation for contrastive learning and pre-fine-tuning of pretrained masked language models confirmed the effectiveness of the proposed model. We release our paraphrase generation model and a corpus of 87 million diverse paraphrases. (https://github.com/Ogamon958/ConPGS)

Monolingual word alignment is crucial to model semantic interactions between sentences.In particular, null alignment, a phenomenon in which words have no corresponding counterparts, is pervasive and critical in handling semantically divergent sentences.Identification of null alignment is useful on its own to reason about the semantic similarity of sentences by indicating there exists information inequality.To achieve unbalanced word alignment that values both alignment and null alignment, this study shows that the family of optimal transport (OT), i.e., balanced, partial, and unbalanced OT, are natural and powerful approaches even without tailor-made techniques.Our extensive experiments covering unsupervised and supervised settings indicate that our generic OT-based alignment methods are competitive against the state-of-the-arts specially designed for word alignment, remarkably on challenging datasets with high null alignment frequencies.

We propose a method that ensembles N -best hypotheses to improve natural language generation.Previous studies have achieved notable improvements in generation quality by explicitly reranking N -best candidates.These studies assume that there exists a hypothesis of higher quality.We expand the assumption to be more practical as there exist partly higher quality hypotheses in the N -best yet they may be imperfect as the entire sentences.By merging these high-quality fragments, we can obtain a higher-quality output than the single-best sentence.Specifically, we first obtain N -best hypotheses and conduct token-level quality estimation.We then apply tokens that should or should not be present in the final output as lexical constraints in decoding.Empirical experiments on paraphrase generation, summarisation, and constrained text generation confirm that our method outperforms the strong N -best reranking methods.

We propose a method to distill language-agnostic meaning embeddings from multilingual sentence encoders for unsupervised quality estimation of machine translation. Our method facilitates that the meaning embeddings focus on semantics by adversarial training that attempts to eliminate language-specific information. Experimental results on unsupervised quality estimation reveal that our method achieved higher correlations with human evaluations.

Controllable text simplification is a crucial assistive technique for language learning and teaching.One of the primary factors hindering its advancement is the lack of a corpus annotated with sentence difficulty levels based on language ability descriptions.To address this problem, we created the CEFR-based Sentence Profile (CEFR-SP) corpus, containing 17k English sentences annotated with the levels based on the Common European Framework of Reference for Languages assigned by English-education professionals.In addition, we propose a sentence-level assessment model to handle unbalanced level distribution because the most basic and highly proficient sentences are naturally scarce.In the experiments in this study, our method achieved a macro-F1 score of 84.5% in the level assessment, thus outperforming strong baselines employed in readability assessment.

This study investigated and released the JADE, a corpus for Japanese definition modelling, which is a technique that automatically generates definitions of a given target word and phrase. It is a crucial technique for practical applications that assist language learning and education, as well as for those supporting reading documents in unfamiliar domains. Although corpora for development of definition modelling techniques have been actively created, their languages are mostly limited to English. In this study, a corpus for Japanese, named JADE, was created following the previous study that mines an online encyclopedia. The JADE provides about 630k sets of targets, their definitions, and usage examples as contexts for about 41k unique targets, which is sufficiently large to train neural models. The targets are both words and phrases, and the coverage of domains and topics is diverse. The performance of a pre-trained sequence-to-sequence model and the state-of-the-art definition modelling method was also benchmarked on JADE for future development of the technique in Japanese. The JADE corpus has been released and available online.

Definition generation techniques aim to generate a definition of a target word or phrase given a context. In previous studies, researchers have faced various issues such as the out-of-vocabulary problem and over/under-specificity problems. Over-specific definitions present narrow word meanings, whereas under-specific definitions present general and context-insensitive meanings. Herein, we propose a method for definition generation with appropriate specificity. The proposed method addresses the aforementioned problems by leveraging a pre-trained encoder-decoder model, namely Text-to-Text Transfer Transformer, and introducing a re-ranking mechanism to model specificity in definitions. Experimental results on standard evaluation datasets indicate that our method significantly outperforms the previous state-of-the-art method. Moreover, manual evaluation confirms that our method effectively addresses the over/under-specificity problems.

We propose a method to distil languageagnostic meaning embedding using a multilingual sentence encoder.By removing languagespecific information from the original embedding, we retrieve an embedding that fully represents the meaning of the sentence.The proposed method relies only on parallel corpora without any human annotations.Our meaning embedding allows for efficient cross-lingual sentence similarity estimation using a simple cosine similarity calculation.Experimental results of both the quality estimation of machine translation and cross-lingual semantic textual similarity tasks reveal that our method consistently outperforms the strong baselines using the original multilingual embeddings.The method also consistently improves the performance of any pre-trained multilingual sentence encoder, even in low-resource language pairs, where only tens of thousands of parallel sentence pairs are available.1

Sentence pair modelling is defined as the task of identifying the semantic interaction between a sentence pair, i.e., paraphrase and textual entailment identification and semantic similarity measurement. It constitutes a set of crucial tasks for research in the area of natural language understanding. Sentence representation learning is a fundamental technology for sentence pair modelling, where the development of the BERT model realised a breakthrough. We have recently proposed transfer fine-tuning using phrasal paraphrases to allow BERT’s representations to be suitable for semantic equivalence assessment between sentences while maintaining the model size. Herein, we reveal that transfer fine-tuning with simplified feature generation allows us to generate representations that are widely effective across different types of sentence pair modelling tasks. Detailed analysis confirms that our transfer fine-tuning helps the BERT model converge more quickly with a smaller corpus for fine-tuning.

The difference in word orders between source and target languages is a serious hurdle for machine translation. Preordering methods, which reorder the words in a source sentence before translation to obtain a similar word ordering with a target language, significantly improve the quality in statistical machine translation. While the information on the preordering position improved the translation quality in recurrent neural network-based models, questions such as how to use preordering information and whether it is helpful for the Transformer model remain unaddressed. In this article, we successfully employed preordering techniques in the Transformer-based neural machine translation. Specifically, we proposed a novel preordering encoding that exploits the reordering information of the source and target sentences as positional encoding in the Transformer model. Experimental results on ASPEC Japanese-English and WMT 2015 English-German, English-Czech, and English-Russian translation tasks confirmed that the proposed method significantly improved the translation quality evaluated by the BLEU scores of the Transformer model by 1.34 points in the Japanese-to-English task, 2.19 points in the English-to-German task, 0.15 points in the Czech-to-English task, and 1.48 points in the English-to-Russian task.

Advanced pre-trained models for text representation have achieved state-of-the-art performance on various text classification tasks.However, the discrepancy between the semantic similarity of texts and labelling standards affects classifiers, i.e. leading to lower performance in cases where classifiers should assign different labels to semantically similar texts.To address this problem, we propose a simple multitask learning model that uses negative supervision.Specifically, our model encourages texts with different labels to have distinct representations.Comprehensive experiments show that our model outperforms the stateof-the-art pre-trained model on both singleand multi-label classifications, sentence and document classifications, and classifications in three different languages.

We reduce the model size of pre-trained word embeddings by a factor of 200 while preserving its quality.Previous studies in this direction created a smaller word embedding model by reconstructing pre-trained word representations from those of subwords, which allows to store only a smaller number of subword embeddings in the memory.However, previous studies that train the reconstruction models using only target words cannot reduce the model size extremely while preserving its quality.Inspired by the observation of words with similar meanings having similar embeddings, our reconstruction training learns the global relationships among words, which can be employed in various models for word embedding reconstruction.Experimental results on word similarity benchmarks show that the proposed method improves the performance of the all subword-based reconstruction models.

We performed a detailed error analysis in domain-specific neural machine translation (NMT) for the English and Japanese language pair with fine-grained manual annotation. Despite its importance for advancing NMT technologies, research on the performance of domain-specific NMT and non-European languages has been limited. In this study, we designed an error typology based on the error types that were typically generated by NMT systems and might cause significant impact in technical translations: “Addition,” “Omission,” “Mistranslation,” “Grammar,” and “Terminology.” The error annotation was targeted to the medical domain and was performed by experienced professional translators specialized in medicine under careful quality control. The annotation detected 4,912 errors on 2,480 sentences, and the frequency and distribution of errors were analyzed. We found that the major errors in NMT were “Mistranslation” and “Terminology” rather than “Addition” and “Omission,” which have been reported as typical problems of NMT. Interestingly, more errors occurred in documents for professionals compared with those for the general public. The results of our annotation work will be published as a parallel corpus with error labels, which are expected to contribute to developing better NMT models, automatic evaluation metrics, and quality estimation models.

Phrase alignment is the basis for modelling sentence pair interactions, such as paraphrase and textual entailment recognition.Most phrase alignments are compositional processes such that an alignment of a phrase pair is constructed based on the alignments of their child phrases.Nonetheless, studies have revealed that non-compositional alignments involving long-distance phrase reordering are prevalent in practice.We address the phrase alignment problem by combining an unordered tree mapping algorithm and phrase representation modelling that explicitly embeds the similarity distribution in the sentences onto powerful contextualized representations.Experimental results demonstrate that our method effectively handles compositional and non-compositional global phrase alignments.Our method significantly outperforms that used in a previous study and achieves a performance competitive with that of experienced human annotators.

Adverse drug reactions are a severe problem that significantly degrade quality of life, or even threaten the life of patients. Patient-generated texts available on the web have been gaining attention as a promising source of information in this regard. While previous studies annotated such patient-generated content, they only reported on limited information, such as whether a text described an adverse drug reaction or not. Further, they only annotated short texts of a few sentences crawled from online forums and social networking services. The dataset we present in this paper is unique for the richness of annotated information, including detailed descriptions of drug reactions with full context. We crawled patient’s weblog articles shared on an online patient-networking platform and annotated the effects of drugs therein reported. We identified spans describing drug reactions and assigned labels for related drug names, standard codes for the symptoms of the reactions, and types of effects. As a first dataset, we annotated 677 drug reactions with these detailed labels based on 169 weblog articles by Japanese lung cancer patients. Our annotation dataset is made publicly available at our web site (https://yukiar.github.io/adr-jp/) for further research on the detection of adverse drug reactions and more broadly, on patient-generated text processing.

We present SAPPHIRE, a Simple Aligner for Phrasal Paraphrase with HIerarchical REpresentation. Monolingual phrase alignment is a fundamental problem in natural language understanding and also a crucial technique in various applications such as natural language inference and semantic textual similarity assessment. Previous methods for monolingual phrase alignment are language-resource intensive; they require large-scale synonym/paraphrase lexica and high-quality parsers. Different from them, SAPPHIRE depends only on a monolingual corpus to train word embeddings. Therefore, it is easily transferable to specific domains and different languages. Specifically, SAPPHIRE first obtains word alignments using pre-trained word embeddings and then expands them to phrase alignments by bilingual phrase extraction methods. To estimate the likelihood of phrase alignments, SAPPHIRE uses phrase embeddings that are hierarchically composed of word embeddings. Finally, SAPPHIRE searches for a set of consistent phrase alignments on a lattice of phrase alignment candidates. It achieves search-efficiency by constraining the lattice so that all the paths go through a phrase alignment pair with the highest alignment score. Experimental results using the standard dataset for phrase alignment evaluation show that SAPPHIRE outperforms the previous method and establishes the state-of-the-art performance.

Yuki Arase, Jun’ichi Tsujii. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

We propose an efficient method to conduct phrase alignment on parse forests for paraphrase detection.Unlike previous studies, our method identifies syntactic paraphrases under linguistically motivated grammar.In addition, it allows phrases to non-compositionally align to handle paraphrases with non-homographic phrase correspondences.A dataset that provides gold parse trees and their phrase alignments is created.The experimental results confirm that the proposed method conducts highly accurate phrase alignment compared to human performance.

Word reordering is a crucial technique in statistical machine translation in which syntactic information plays an important role.Synchronous context-free grammar has typically been used for this purpose with various modifications for adding flexibilities to its synchronized tree generation.We permit further flexibilities in the synchronous context-free grammar in order to translate between languages with drastically different word order.Our method pre-processes a parallel corpus by abstracting source-side dependency trees, and performs long-distance reordering on top of an off-the-shelf phrase-based system.Experimental results show that our method significantly outperforms previous phrase-based and syntax-based models for translation between English and Japanese.

A variety of services utilizing users' positions have become available because of rapid advances in Global Positioning System (GPS) technologies. Since location information may reveal private information, preserving location privacy has become a significant issue. We proposed a dummy-based method of anonymizing location to protect this privacy in our previous work that generated dummies based on various restrictions in a real environment. However, the previous work assumed a simplified mobility model in which users kept moving and did not stop. If we assume a more realistic mobility model in which users often pause to visit various attractions, it becomes increasingly more difficult to generate dummies that will move naturally. In this paper, we assumed that the users' movements are known in advance and propose a dummy-based anonymization method based on user movements, where dummies move naturally while stopping at several locations. We simulated user movements on real map information and verified the method we propose was more effective than the previous one.

Recent mobile devices are mostly equipped with a GPS receiver. This trend has arisen a variety of location based services (LBSs) which enable users to search local information of their current locations. To use LBSs, a user needs to send his/her location information to a service provider. Users' location information is inherently private, since it can reveal the critical information. In this paper, we propose a method to protect the user's location privacy by sending the user's location with dummy locations, which are determined based on the user's current location. We conduct an experiment to evaluate the effectiveness of our proposed method using real users' trajectories on a real map.

Photo sharing is one of the most popular Web services. Photo sharing sites provide functions to add tags and geo-tags to photos to make photo organization easy. Considering that people take photos to record something that attracts them, geo-tagged photos are a rich data source that reflects people's memorable events associated with locations. In this paper, we focus on geo-tagged photos and propose a method to detect people's frequent trip patterns, i.e., typical sequences of visited cities and durations of stay as well as descriptive tags that characterize the trip patterns. Our method first segments photo collections into trips and categorizes them based on their trip themes, such as visiting landmarks or communing with nature. Our method mines frequent trip patterns for each trip theme category. We crawled 5.7 million geo-tagged photos and performed photo trip pattern mining. The experimental result shows that our method outperforms other baseline methods and can correctly segment photo collections into photo trips with an accuracy of 78%. For trip categorization, our method can categorize about 80% of trips using tags and titles of photos and visited cities as features. Finally, we illustrate interesting examples of trip patterns detected from our dataset and show an application with which users can search frequent trip patterns by querying a destination, visit duration, and trip theme on the trip.

Cellular phones are widely used to access the Web. However, because of their small screens and poor interfaces, it is inconvenient to browse Web pages designed for desktop PCs. In this paper, we propose two novel functions that virtually extend a cellular phone's screen so that users can acquire more information from the small screens than the traditional Web browsing using cellular phones. The first function presents previews of adjacent areas of the currently displayed area of a Web page so that users can understand what information is in the neighboring areas. The other presents previews of Web pages linked from the current page to help users to select an appropriate link connecting to a page containing the information of interest. As a platform for the implementation of these functions, we use cellular phones equipped with sensors to provide a user-friendly interface.

Cellular phones are widely used to access the Web, however, it is inconvenient to browse large-sized pages designed for desktop PCs. Since cellular phones display only small part of a Web page, users easily get lost within the page while scrolling around. Even though pages are well structured by nature, users cannot recognize it because visible portion is strictly limited. To smartly browse pages using cellular phones, navigation is essential. In this paper, we propose a novel Web browser that models browsing as motoring cities, by setting routes on Web pages and presenting traffic signs to navigate users. Users can follow routes by auto-scrolling without burdensome operations.

Geographical context is very important for images. Millions of images on the Web have been already assigned latitude and longitude information. Due to the rapid proliferation of such images with geographical context, it is still difficult to effectively search and browse them, since we do not have ways to decide their relevance. In this paper, we focus on the geographical relevance of images, which is defined as to what extent the main objects in an image match landmarks at the location where the image was taken. Recently, researchers have proposed to use game based approaches to label large scale data such as Web images. However, previous works have not shown the quality of collected game logs in detail and how the logs can improve existing applications. To answer these questions, we design and implement a Web-based and multi-player game to collect human knowledge while people are enjoying the game. Then we thoroughly analyze the game logs obtained during a three week study with 147 participants and propose methods to determine the image geographical relevance. In addition, we conduct an experiment to compare our methods with a commercial search engine. Experimental results show that our methods dramatically improve image search relevance. Furthermore, we show that we can derive geographically relevant objects and their salient portion in images, which is valuable for a number of applications such as image location recognition.

Cellular phones are widely used to access the WWW. However, most available Web pages are designed for desktop PCs. Cellular phones only have small screens and poor interfaces, and thus, it is inconvenient to browse such large sized pages. In addition, cellular phone users browse Web pages in various situations, so that appropriate presentation styles for Web pages depend on users' situations. In this paper, we propose a novel Web browsing system for cellular phones that allocates various functions for Web browsing on each numerical key of a cellular phone. Users can browse Web pages comfortably, selecting appropriate functions according to their situations by pushing a single button.